Device, method, and graphical user interface for providing tactile feedback for media adjustment operations performed in a user interface

ABSTRACT

An electronic device with a touch-sensitive surface and a display displays content that includes an image, detects a gesture on the touch-sensitive surface for adjusting an image size of the image, and in response to detecting the gesture, determines an adjustment to the image size that corresponds to an extent of the gesture. If the adjustment to the image size would cause the image size to exceed a predefined image sizing limit, the device generates a respective tactile output on the touch-sensitive surface, performs the adjustment to the image size, and in response to detecting liftoff of a contact used to perform the gesture, at least partially reverses the adjustment to the image size, such that the image size is reduced to a size corresponding to the predefined image sizing limit.

RELATED APPLICATIONS

This application is a Continuation of PCT Patent Application Serial No.PCT/US2013/040070, filed on May 8, 2013, entitled “Device, Method, andGraphical User Interface for Providing Tactile Feedback for OperationsPerformed in a User Interface,” which claims the benefit of and priorityto U.S. Provisional Patent Application Ser. No. 61/778,284, filed onMar. 12, 2013, entitled “Device, Method, and Graphical User Interfacefor Providing Tactile Feedback for Operations Performed in a UserInterface;” U.S. Provisional Patent Application No. 61/747,278, filedDec. 29, 2012, entitled “Device, Method, and Graphical User Interfacefor Manipulating User Interface Objects with Visual and/or HapticFeedback;” and U.S. Provisional Patent Application No. 61/688,227, filedMay 9, 2012, entitled “Device, Method, and Graphical User Interface forManipulating User Interface Objects with Visual and/or Haptic Feedback,”which applications are incorporated by reference herein in theirentireties.

This application is also related to the following: U.S. ProvisionalPatent Application Ser. No. 61/778,092, filed on Mar. 12, 2013, entitled“Device, Method, and Graphical User Interface for Selecting Objectwithin a Group of Objects;” U.S. Provisional Patent Application Ser. No.61/778,125, filed on Mar. 12, 2013, entitled “Device, Method, andGraphical User Interface for Navigating User Interface Hierarchies;”U.S. Provisional Patent Application Ser. No. 61/778,156, filed on Mar.12, 2013, entitled “Device, Method, and Graphical User Interface forManipulating Framed Graphical Objects;” U.S. Provisional PatentApplication Ser. No. 61/778,179, filed on Mar. 12, 2013, entitled“Device, Method, and Graphical User Interface for Scrolling NestedRegions;” U.S. Provisional Patent Application Ser. No. 61/778,171, filedon Mar. 12, 2013, entitled “Device, Method, and Graphical User Interfacefor Displaying Additional Information in Response to a User Contact;”U.S. Provisional Patent Application Ser. No. 61/778,191, filed on Mar.12, 2013, entitled “Device, Method, and Graphical User Interface forDisplaying User Interface Objects Corresponding to an Application;” U.S.Provisional Patent Application Ser. No. 61/778,211, filed on Mar. 12,2013, entitled “Device, Method, and Graphical User Interface forFacilitating User Interaction with Controls in a User Interface;” U.S.Provisional Patent Application Ser. No. 61/778,239, filed on Mar. 12,2013, entitled “Device, Method, and Graphical User Interface forForgoing Generation of Tactile Output for a Multi-Contact Gesture;” U.S.Provisional Patent Application Ser. No. 61/778,287, filed on Mar. 12,2013, entitled “Device, Method, and Graphical User Interface forProviding Feedback for Changing Activation States of a User InterfaceObject;” U.S. Provisional Patent Application Ser. No. 61/778,363, filedon Mar. 12, 2013, entitled “Device, Method, and Graphical User Interfacefor Transitioning between Touch Input to Display Output Relationships;”U.S. Provisional Patent Application Ser. No. 61/778,367, filed on Mar.12, 2013, entitled “Device, Method, and Graphical User Interface forMoving a User Interface Object Based on an Intensity of a Press Input;”U.S. Provisional Patent Application Ser. No. 61/778,265, filed on Mar.12, 2013, entitled “Device, Method, and Graphical User Interface forTransitioning between Display States in Response to a Gesture;” U.S.Provisional Patent Application Ser. No. 61/778,373, filed on Mar. 12,2013, entitled “Device, Method, and Graphical User Interface forManaging Activation of a Control Based on Contact Intensity;” U.S.Provisional Patent Application Ser. No. 61/778,412, filed on Mar. 13,2013, entitled “Device, Method, and Graphical User Interface forDisplaying Content Associated with a Corresponding Affordance;” U.S.Provisional Patent Application Ser. No. 61/778,413, filed on Mar. 13,2013, entitled “Device, Method, and Graphical User Interface forSelecting User Interface Objects;” U.S. Provisional Patent ApplicationSer. No. 61/778,414, filed on Mar. 13, 2013, entitled “Device, Method,and Graphical User Interface for Moving and Dropping a User InterfaceObject;” U.S. Provisional Patent Application Ser. No. 61/778,416, filedon Mar. 13, 2013, entitled “Device, Method, and Graphical User Interfacefor Determining Whether to Scroll or Select Content;” and U.S.Provisional Patent Application Ser. No. 61/778,418, filed on Mar. 13,2013, entitled “Device, Method, and Graphical User Interface forSwitching between User Interfaces,” which are incorporated herein byreference in their entireties.

This application is also related to the following: U.S. ProvisionalPatent Application Ser. No. 61/645,033, filed on May 9, 2012, entitled“Adaptive Haptic Feedback for Electronic Devices;” U.S. ProvisionalPatent Application Ser. No. 61/665,603, filed on Jun. 28, 2012, entitled“Adaptive Haptic Feedback for Electronic Devices;” and U.S. ProvisionalPatent Application Ser. No. 61/681,098, filed on Aug. 8, 2012, entitled“Adaptive Haptic Feedback for Electronic Devices,” which areincorporated herein by reference in their entireties.

TECHNICAL FIELD

This relates generally to electronic devices with touch-sensitivesurfaces, including but not limited to electronic devices withtouch-sensitive surfaces that detect inputs for manipulating userinterfaces.

BACKGROUND

The use of touch-sensitive surfaces as input devices for computers andother electronic computing devices has increased significantly in recentyears. Exemplary touch-sensitive surfaces include touch pads and touchscreen displays. Such surfaces are widely used to manipulate userinterface objects on a display.

Exemplary manipulations include adjusting the position and/or size ofone or more user interface objects or activating buttons or openingfiles/applications represented by user interface objects, as well asassociating metadata with one or more user interface objects orotherwise manipulating user interfaces. Exemplary user interface objectsinclude digital images, video, text, icons, control elements such asbuttons and other graphics. A user will, in some circumstances, need toperform such manipulations on user interface objects in a filemanagement program (e.g., Finder from Apple Inc. of Cupertino, Calif.),an image management application (e.g., Aperture or iPhoto from AppleInc. of Cupertino, Calif.), a digital content (e.g., videos and music)management application (e.g., iTunes from Apple Inc. of Cupertino,Calif.), a drawing application, a presentation application (e.g.,Keynote from Apple Inc. of Cupertino, Calif.), a word processingapplication (e.g., Pages from Apple Inc. of Cupertino, Calif.), awebsite creation application (e.g., iWeb from Apple Inc. of Cupertino,Calif.), a disk authoring application (e.g., iDVD from Apple Inc. ofCupertino, Calif.), or a spreadsheet application (e.g., Numbers fromApple Inc. of Cupertino, Calif.).

But existing methods for performing these manipulations are cumbersomeand inefficient. In addition, existing methods take longer thannecessary, thereby wasting energy. This latter consideration isparticularly important in battery-operated devices.

SUMMARY

Accordingly, there is a need for electronic devices with faster, moreefficient methods and interfaces for manipulating user interfaces. Suchmethods and interfaces optionally complement or replace conventionalmethods for manipulating user interfaces. Such methods and interfacesreduce the cognitive burden on a user and produce a more efficienthuman-machine interface. For battery-operated devices, such methods andinterfaces conserve power and increase the time between battery charges.

The above deficiencies and other problems associated with userinterfaces for electronic devices with touch-sensitive surfaces arereduced or eliminated by the disclosed devices. In some embodiments, thedevice is a desktop computer. In some embodiments, the device isportable (e.g., a notebook computer, tablet computer, or handhelddevice). In some embodiments, the device has a touchpad. In someembodiments, the device has a touch-sensitive display (also known as a“touch screen” or “touch screen display”). In some embodiments, thedevice has a graphical user interface (GUI), one or more processors,memory and one or more modules, programs or sets of instructions storedin the memory for performing multiple functions. In some embodiments,the user interacts with the GUI primarily through finger contacts andgestures on the touch-sensitive surface. In some embodiments, thefunctions optionally include image editing, drawing, presenting, wordprocessing, website creating, disk authoring, spreadsheet making, gameplaying, telephoning, video conferencing, e-mailing, instant messaging,workout support, digital photographing, digital videoing, web browsing,digital music playing, and/or digital video playing. Executableinstructions for performing these functions are, optionally, included ina non-transitory computer readable storage medium or other computerprogram product configured for execution by one or more processors.

There is a need for electronic devices with more methods and interfacesfor providing tactile feedback for operations performed in a userinterface. Such methods and interfaces may complement or replaceconventional methods for providing feedback for operations performed ina user interface. Such methods and interfaces reduce the cognitiveburden on a user and produce a more efficient human-machine interface.

In accordance with some embodiments, a method is performed at anelectronic device with a display and a touch-sensitive surface. Themethod includes displaying a user interface object on the display,detecting a contact on the touch-sensitive surface, and detecting afirst movement of the contact across the touch-sensitive surface, thefirst movement corresponding to performing an operation on the userinterface object, and, in response to detecting the first movement,performing the operation and generating a first tactile output on thetouch-sensitive surface. The method further includes detecting a secondmovement of the contact across the touch-sensitive surface, the secondmovement corresponding to reversing the operation on the user interfaceobject, and in response to detecting the second movement, reversing theoperation and generating a second tactile output on the touch-sensitivesurface, wherein the second tactile output is different from the firsttactile output.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display a user interface object, atouch-sensitive surface unit configured to detect user contacts, and aprocessing unit coupled to the display unit and the touch-sensitivesurface unit. The processing unit is configured to detect a contact onthe touch-sensitive surface unit, detect a first movement of the contactacross the touch-sensitive surface unit, the first movementcorresponding to performing an operation on the user interface object,in response to detecting the first movement; perform the operation andgenerate a first tactile output on the touch-sensitive surface unit. Theprocessing unit is further configured to detect a second movement of thecontact across the touch-sensitive surface unit, the second movementcorresponding to reversing the operation on the user interface object,and in response to detecting the second movement; reverse the operationand generate a second tactile output on the touch-sensitive surfaceunit, where the second tactile output is different from the firsttactile output.

Thus, electronic devices with displays and touch-sensitive surfaces areprovided with more methods and interfaces for providing tactile feedbackfor operations performed in a user interface, thereby increasing theeffectiveness, efficiency, and user satisfaction with such devices. Suchmethods and interfaces may complement or replace conventional methodsfor providing feedback for operations performed in a user interface.

There is a need for electronic devices with faster, more efficientmethods and interfaces for indicating changes in the z-order of userinterface objects. Such methods and interfaces may complement or replaceconventional methods for indicating changes in the z-order of userinterface objects. Such methods and interfaces reduce the cognitiveburden on a user and produce a more efficient human-machine interface.For battery-operated devices, such methods and interfaces conserve powerand increase the time between battery charges.

In accordance with some embodiments, a method is performed at anelectronic device with a display and a touch-sensitive surface. Themethod includes: displaying a plurality of user interface objects on thedisplay, where: the plurality of user interface objects have a z-order,the plurality of user interface objects includes a first user interfaceobject and a second user interface object, and the first user interfaceobject is above the second user interface object in the z-order; whiledetecting a contact on the touch-sensitive surface, receiving a requestto move the first user interface object below the second user interfaceobject in the z-order; and in response to the request: moving the firstuser interface object below the second user interface object in thez-order; in accordance with a determination that the first userinterface object overlaps at least a portion of the second userinterface object, generating a tactile output associated with moving thefirst user interface object below the second user interface object onthe touch-sensitive surface in conjunction with moving the first userinterface object below the second user interface object; and inaccordance with a determination that the first user interface objectdoes not overlap the second user interface object, forgoing generatingthe tactile output associated with moving the first user interfaceobject below the second user interface object.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display a plurality of user interface objectson the display unit, where: the plurality of user interface objects havea z-order, the plurality of user interface objects includes a first userinterface object and a second user interface object, and the first userinterface object is above the second user interface object in thez-order; a touch-sensitive surface unit configured to receive contacts;and a processing unit coupled to the display unit and thetouch-sensitive surface unit. The processing unit is configured to:while detecting a contact on the touch-sensitive surface unit, receive arequest to move the first user interface object below the second userinterface object in the z-order; and in response to the request: movethe first user interface object below the second user interface objectin the z-order; in accordance with a determination that the first userinterface object overlaps at least a portion of the second userinterface object, generate a tactile output associated with moving thefirst user interface object below the second user interface object onthe touch-sensitive surface unit in conjunction with moving the firstuser interface object below the second user interface object; and inaccordance with a determination that the first user interface objectdoes not overlap the second user interface object, forgo generating thetactile output associated with moving the first user interface objectbelow the second user interface object.

Thus, electronic devices with displays and touch-sensitive surfaces areprovided with faster, more efficient methods and interfaces forindicating changes in the z-order of user interface objects, therebyincreasing the effectiveness, efficiency, and user satisfaction withsuch devices. Such methods and interfaces may complement or replaceconventional methods for indicating changes in the z-order of userinterface objects.

There is a need for electronic devices with faster, more efficientmethods and interfaces for providing feedback when an action will resultin the adjustment of a parameter beyond a predefined limit. Such methodsand interfaces may complement or replace conventional methods forproviding feedback when an action will result in the adjustment of aparameter beyond a predefined limit. Such methods and interfaces reducethe cognitive burden on a user and produce a more efficienthuman-machine interface. For battery-operated devices, such methods andinterfaces conserve power and increase the time between battery charges.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface. The methodincludes: displaying, on the display, a control for controlling aparameter associated with respective content. The method furtherincludes: detecting a gesture on the touch-sensitive surface foradjusting the parameter. The method further includes, in response todetecting the gesture: determining an adjustment of the parameter thatcorresponds to an extent of the gesture; in accordance with adetermination that the adjustment of the parameter would cause one ormore predefined adjustment limits to be exceeded, generating arespective tactile output on the touch-sensitive surface; and inaccordance with a determination that the adjustment of the parameterwould not cause the one or more predefined adjustment limits to beexceeded, performing the adjustment of the parameter without generatingthe respective tactile output on the touch-sensitive surface.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display a control for controlling a parameterassociated with respective content; a touch-sensitive surface unitconfigured to receive user contacts; and a processing unit coupled tothe display unit and the touch-sensitive surface unit. The processingunit is configured to: enable display of a control for controlling aparameter associated with respective content on the display unit; anddetect a gesture on the touch-sensitive surface unit for adjusting theparameter. The processing unit is further configured to, in response todetecting the gesture: determine an adjustment of the parameter thatcorresponds to an extent of the gesture; in accordance with adetermination that the adjustment of the parameter would cause one ormore predefined adjustment limits to be exceeded, generate a respectivetactile output on the touch-sensitive surface unit; and in accordancewith a determination that the adjustment of the parameter would notcause the one or more predefined adjustment limits to be exceeded,perform the adjustment of the parameter without generating therespective tactile output on the touch-sensitive surface unit.

Thus, electronic devices with displays, touch-sensitive surfaces areprovided with faster, more efficient methods and interfaces forproviding feedback when an action will result in the adjustment of aparameter beyond a predefined limit, thereby increasing theeffectiveness, efficiency, and user satisfaction with such devices. Suchmethods and interfaces may complement or replace conventional methodsfor providing feedback when an action will result in the adjustment of aparameter beyond a predefined limit.

There is a need for electronic devices with more methods and interfacesfor providing feedback corresponding to a clock. Such methods andinterfaces may complement or replace conventional methods for displayinga clock. Such methods and interfaces reduce the cognitive burden on auser and produce a more efficient human-machine interface. Forbattery-operated devices, such methods and interfaces conserve power andincrease the time between battery charges.

In accordance with some embodiments, a method is performed at anelectronic device with a display and a touch-sensitive surface. Themethod includes displaying a representation of a clock on the display,detecting movement of a focus selector over the representation of theclock; while detecting the focus selector over the representation of theclock, providing tactile feedback that corresponds to the clock, wherethe tactile feedback includes a regular pattern of tactile outputs onthe touch-sensitive surface. The method further includes, whileproviding the tactile feedback, detecting movement of the focus selectoraway from the representation of the clock, and in response to detectingmovement of the focus selector away from the representation of theclock, ceasing to provide the tactile feedback corresponding to theclock.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display a representation of a clock, atouch-sensitive surface unit, and a processing unit coupled to thedisplay unit and the touch-sensitive surface unit. The processing unitis configured to: detect movement of a focus selector over therepresentation of the clock, while detecting the focus selector over therepresentation of the clock, provide tactile feedback that correspondsto the clock, where the tactile feedback includes a regular pattern oftactile outputs on the touch-sensitive surface unit. The processing unitis further configured to, while providing the tactile feedback, detectmovement of the focus selector away from the representation of theclock, and in response to detecting movement of the focus selector awayfrom the representation of the clock, cease to provide the tactilefeedback corresponding to the clock.

Thus, electronic devices with displays and touch-sensitive surfaces areprovided with more methods and interfaces for providing feedbackcorresponding to a clock, thereby increasing the effectiveness,efficiency, and user satisfaction with such devices. Such methods andinterfaces may complement or replace conventional methods for providingfeedback corresponding to a clock.

There is a need for electronic devices with faster, more efficientmethods and interfaces for providing feedback that corresponds to beatsof a piece of music. Such methods and interfaces may complement orreplace conventional methods for providing feedback that corresponds tobeats of a piece of music. Such methods and interfaces reduce thecognitive burden on a user and produce a more efficient human-machineinterface. For battery-operated devices, such methods and interfacesconserve power and increase the time between battery charges.

In accordance with some embodiments, a method is performed at anelectronic device with a display, a touch-sensitive surface. The methodincludes displaying a representation of a piece of music on the display.The method further includes detecting movement of a focus selector overthe representation of the piece of music. The method further includes,while detecting the focus selector over the representation of the pieceof music, providing tactile feedback that corresponds to at least asubset of beats of the piece of music. The method further includes,after providing the tactile feedback, detecting movement of the focusselector away from the representation of the piece of music. The methodfurther includes, in response to detecting movement of the focusselector away from the representation of the piece of music, ceasing toprovide the tactile feedback that corresponds to the beats of the pieceof music.

In accordance with some embodiments, an electronic device includes adisplay unit configured to display a representation of a piece of music;a touch-sensitive surface unit configured to receive user contacts; anda processing unit coupled to the display unit and the touch-sensitivesurface unit. The processing unit is configured to: enable display of arepresentation of a piece of music; and detect movement of a focusselector over the representation of the piece of music. The processingunit is further configured to, while detecting the focus selector overthe representation of the piece of music, provide tactile feedback thatcorresponds to at least a subset of beats of the piece of music. Theprocessing unit is further configured to, after providing the tactilefeedback, detect movement of the focus selector away from therepresentation of the piece of music. The processing unit is furtherconfigured to, in response to detecting movement of the focus selectoraway from the representation of the piece of music, cease to provide thetactile feedback that corresponds to the beats of the piece of music.

Thus, electronic devices with displays, touch-sensitive surfaces areprovided with faster, more efficient methods and interfaces forproviding feedback that corresponds to beats of a piece of music,thereby increasing the effectiveness, efficiency, and user satisfactionwith such devices. Such methods and interfaces may complement or replaceconventional methods for providing feedback that corresponds to beats ofa piece of music.

In accordance with some embodiments, an electronic device includes adisplay, a touch-sensitive surface, optionally one or more sensors todetect intensity of contacts with the touch-sensitive surface, one ormore processors, memory, and one or more programs; the one or moreprograms are stored in the memory and configured to be executed by theone or more processors and the one or more programs include instructionsfor performing the operations of any of the methods referred. Inaccordance with some embodiments, a graphical user interface on anelectronic device with a display, a touch-sensitive surface, optionallyone or more sensors to detect intensity of contacts with thetouch-sensitive surface, a memory, and one or more processors to executeone or more programs stored in the memory includes one or more of theelements displayed in any of the methods referred, which are updated inresponse to inputs, as described in any of the methods referred. Inaccordance with some embodiments, a computer readable storage medium hasstored therein instructions which when executed by an electronic devicewith a display, a touch-sensitive surface, and optionally one or moresensors to detect intensity of contacts with the touch-sensitivesurface, cause the device to perform the operations of any of themethods referred. In accordance with some embodiments, an electronicdevice includes: a display, a touch-sensitive surface, and optionallyone or more sensors to detect intensity of contacts with thetouch-sensitive surface; and means for performing the operations of anyof the methods referred. In accordance with some embodiments, aninformation processing apparatus, for use in an electronic device with adisplay and a touch-sensitive surface, optionally one or more sensors todetect intensity of contacts with the touch-sensitive surface, includesmeans for performing the operations of any of the methods referred.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on a portable multifunction device in accordance with someembodiments.

FIG. 4B illustrates an exemplary user interface for a multifunctiondevice with a touch-sensitive surface that is separate from the displayin accordance with some embodiments.

FIGS. 5A-5O illustrate exemplary user interfaces for providing tactilefeedback for operations performed in a user interface in accordance withsome embodiments.

FIGS. 6A-6C are flow diagrams illustrating a method of providing tactilefeedback for operations performed in a user interface in accordance withsome embodiments.

FIG. 7 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 8A-8S illustrate exemplary user interfaces for indicating changesin z-order of user interface objects in accordance with someembodiments.

FIGS. 9A-9D are flow diagrams illustrating a method of indicatingchanges in z-order of user interface objects in accordance with someembodiments.

FIG. 10 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 11A-11T illustrate exemplary user interfaces for providingfeedback when an action will result in the adjustment of a parameterbeyond a predefined limit in accordance with some embodiments.

FIGS. 12A-12B are flow diagrams illustrating a method of providingfeedback when an action will result in the adjustment of a parameterbeyond a predefined limit in accordance with some embodiments.

FIG. 13 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 14A-14J illustrate exemplary user interfaces for providing tactilefeedback corresponding to a clock in accordance with some embodiments.

FIGS. 15A-15B are flow diagrams illustrating a method of providingtactile feedback corresponding to a clock in accordance with someembodiments.

FIG. 16 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 17A-17L illustrate exemplary user interfaces for providingfeedback that corresponds to beats of a piece of music in accordancewith some embodiments.

FIGS. 17M-17O illustrate exemplary waveforms of movement profiles forgenerating tactile outputs in accordance with some embodiments.

FIGS. 18A-18B are flow diagrams illustrating a method of providingfeedback that corresponds to beats of a piece of music in accordancewith some embodiments.

FIG. 19 is a functional block diagram of an electronic device inaccordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

The methods, devices and GUIs described herein provide visual and/orhaptic feedback that makes manipulation of user interface objects moreefficient and intuitive for a user. For example, in a system where theclicking action of a trackpad is decoupled from the contact intensity(e.g., contact force, contact pressure, or a substitute therefore) thatis needed to reach an activation threshold, the device can generatedifferent tactile outputs (e.g., “different clicks”) for differentactivation events (e.g., so that clicks that accomplish a particularresult are differentiated from clicks that do not produce any result orthat accomplish a different result from the particular result).Additionally, tactile outputs can be generated in response to otherevents that are not related to increasing intensity of a contact, suchas generating a tactile output (e.g., a “detent”) when a user interfaceobject is moved to a particular position, boundary or orientation, orwhen an event occurs at the device.

Additionally, in a system where a trackpad or touch-screen display issensitive to a range of contact intensity that includes more than one ortwo specific intensity values (e.g., more than a simple on/off, binaryintensity determination), the user interface can provide responses(e.g., visual or tactile cues) that are indicative of the intensity ofthe contact within the range. In some implementations, apre-activation-threshold response and/or a post-activation-thresholdresponse to an input are displayed as continuous animations. As oneexample of such a response, a preview of an operation is displayed inresponse to detecting an increase in contact intensity that is stillbelow an activation threshold for performing the operation. As anotherexample of such a response, an animation associated with an operationcontinues even after the activation threshold for the operation has beenreached. Both of these examples provide a user with a continuousresponse to the force or pressure of a user's contact, which provides auser with visual and/or haptic feedback that is richer and moreintuitive. More specifically, such continuous force responses give theuser the experience of being able to press lightly to preview anoperation and/or press deeply to push “past” or “through” a predefineduser interface state corresponding to the operation.

Additionally, for a device with a touch-sensitive surface that issensitive to a range of contact intensity, multiple contact intensitythresholds can be monitored by the device and different functions can bemapped to different contact intensity thresholds. This serves toincrease the available “gesture space” providing easy access to advancedfeatures for users who know that increasing the intensity of a contactat or beyond a second “deep press” intensity threshold will cause thedevice to perform a different operation from an operation that would beperformed if the intensity of the contact is between a first“activation” intensity threshold and the second “deep press” intensitythreshold. An advantage of assigning additional functionality to asecond “deep press” intensity threshold while maintaining familiarfunctionality at a first “activation” intensity threshold is thatinexperienced users who are, in some circumstances, confused by theadditional functionality can use the familiar functionality by justapplying an intensity up to the first “activation” intensity threshold,whereas more experienced users can take advantage of the additionalfunctionality by applying an intensity at the second “deep press”intensity threshold.

Additionally, for a device with a touch-sensitive surface that issensitive to a range of contact intensity, the device can provideadditional functionality by allowing users to perform complex operationswith a single continuous contact. For example, when selecting a group ofobjects, a user can move a continuous contact around the touch-sensitivesurface and can press while dragging (e.g., applying an intensitygreater than a “deep press” intensity threshold) to add additionalelements to a selection. In this way, a user can intuitively interactwith a user interface where pressing harder with a contact causesobjects in the user interface to be “stickier.”

A number of different approaches to providing an intuitive userinterface on a device where a clicking action is decoupled from theforce that is needed to reach an activation threshold and/or the deviceis sensitive to a wide range of contact intensities are described below.Using one or more of these approaches (optionally in conjunction witheach other) helps to provide a user interface that intuitively providesusers with additional information and functionality, thereby reducingthe user's cognitive burden and improving the human-machine interface.Such improvements in the human-machine interface enable users to use thedevice faster and more efficiently. For battery-operated devices, theseimprovements conserve power and increase the time between batterycharges. For ease of explanation, systems, methods and user interfacesfor including illustrative examples of some of these approaches aredescribed below, as follows:

-   -   Many electronic devices have graphical user interfaces that        include user interface objects. There are usually many        operations which can be performed on the interface objects. For        example, an interface object can be snapped to a guideline or        removed from a guideline. Some user interfaces provide visual        feedback indicating whether an operation has been performed or        reversed. However, in some situations, a user will not notice        the visual feedback and thus will be confused as to whether the        operation has been performed or reversed. The embodiments        described below improve on these methods by generating tactile        outputs for the user corresponding to the operations performed,        thereby providing a more convenient and efficient user        interface. In particular, FIGS. 5A-5O illustrate exemplary user        interfaces for providing tactile feedback for operations        performed in a user interface. FIGS. 6A-6C are flow diagrams        illustrating a method of providing tactile feedback for        operations performed in a user interface. The user interfaces in        FIGS. 5A-5O are further used to illustrate the processes        described below with reference to FIGS. 6A-6C.    -   Many electronic devices display user interface objects that have        a layer order (e.g., a z-order or front-to-back order of the        user interface objects). In some circumstances, a user interacts        with such objects by repositioning them on the display, and        overlapping objects are displayed on the display in accordance        with their front-to-back order (e.g., an object that is “in        front” of another object is displayed where the two objects        overlap). In addition to repositioning the objects on the        display, a user often wants to change the front-to-back order of        the objects on the display. In some methods, changes in the        z-order are indicated with visual feedback. However, in some        situations, a user will not notice the visual feedback and thus        will be confused as to whether the operation has been performed.        The embodiments described below improve on these methods by        providing for tactile outputs when objects overlap each other        and their z-order changes, thereby providing a more convenient        and efficient user interface. In particular, FIGS. 8A-8S        illustrate exemplary user interfaces for indicating changes in        the z-order of user interface objects. FIGS. 9A-9D are flow        diagrams illustrating a method of indicating changes in the        z-order of user interface objects. The user interfaces in FIGS.        8A-8S are used to illustrate the processes in FIGS. 9A-9D.    -   Many electronic devices have graphical user interfaces that        display user interface objects that can be manipulated by        adjusting one or more associated parameter such as the size of a        user interface object. For practical reasons, some of these        parameters have predefined adjustment limits are commonly        assigned to these user interface object, limiting the extent to        which their properties can be adjusted. Some user interfaces        provide visual feedback indicating whether a predefined        adjustment limit has been exceeded. However, in some situations,        a user will not notice the visual feedback and thus will be        confused as to whether or not the predefined adjustment limit        has been exceeded. The embodiments described below provide        improved methods and user interfaces for generating feedback to        a user navigating a complex user interface by providing tactile        feedback when an action will result in the adjustment of a        parameter beyond a predefined adjustment limit, thereby        providing a more convenient and efficient user interface. In        particular, FIGS. 11A-11T illustrate exemplary user interfaces        for providing feedback when an action will result in the        adjustment of a parameter beyond a predefined limit. FIGS.        12A-12B are flow diagrams illustrating a method of providing        feedback when an action will result in the adjustment of a        parameter beyond a predefined limit. The user interfaces in        FIGS. 11A-11T are used to illustrate the processes in FIGS.        12A-12B.    -   Many electronic devices have graphical user interfaces that        include a representation of a clock. There is often a need to        provide efficient and convenient ways for users to receive        feedback corresponding to the clock. Some user interfaces        provide visual feedback indicating advancement of time on a        clock. However, in some situations, a user will look away from        the clock or be distracted and will not be able to pay attention        to the visual feedback while performing another task. The        embodiments below improve on the these methods by generating        tactile outputs for the user that correspond to the clock (e.g.,        a ‘tick tock’ pattern of tactile outputs), thereby providing a        more convenient and efficient user interface by enabling the        user to pay attention to different visual element while        monitoring the advancement of time on the clock. In particular,        FIGS. 14A-14J illustrate exemplary user interfaces for providing        tactile feedback corresponding to a clock in accordance with        some embodiments. The user interfaces in these figures are used        to illustrate the processes described below, including the        processes described below with reference to FIGS. 15A-15B.    -   Many electronic devices have graphical user interfaces that        display application windows showing representations of a piece        of music (e.g., a graphical representation of a piece of cover        art for an album of the piece of music, a region indicating that        a piece of music is being currently being played or notes of a        piece of music in a graphical representation of a music score        corresponding to a piece of music). Given the complexity of user        interface environment that includes application windows        corresponding to applications having both audio and visual        components (e.g., music playback, music composition, video        playback or video composition applications), there is a need to        provide feedback that enables the user to more efficiently and        conveniently navigate through the user interface environment.        Some user interfaces provide visual feedback indicating notes of        a piece of music. However, in some situations, a user will look        away from the region of the user interface providing visual        feedback indicating notes of a piece of music or be distracted        and will not be able to pay attention to the visual feedback        while performing another task. The embodiments described below        provide improved methods and user interfaces for generating        feedback to a user navigating a complex user interface        environment by generating tactile outputs corresponding to notes        in a piece of music, thereby providing a more convenient and        efficient user interface by enabling the user to pay attention        to different visual element while monitoring the notes in the        piece of music. More specifically, these methods and user        interfaces provide feedback that corresponds to beats of a piece        of music represented on a display. Below, FIGS. 17A-17L        illustrate exemplary user interfaces for providing feedback that        corresponds to beats of a piece of music. FIGS. 18A-18B are flow        diagrams illustrating a method of providing feedback that        corresponds to beats of a piece of music. The user interfaces in        FIGS. 17A-17L are used to illustrate the processes in FIGS.        18A-18B.

Exemplary Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Exemplary embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch screen displays and/or touch pads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch screendisplay and/or a touch pad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a drawing application, a presentationapplication, a word processing application, a website creationapplication, a disk authoring application, a spreadsheet application, agaming application, a telephone application, a video conferencingapplication, an e-mail application, an instant messaging application, aworkout support application, a photo management application, a digitalcamera application, a digital video camera application, a web browsingapplication, a digital music player application, and/or a digital videoplayer application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive displays 112 inaccordance with some embodiments. Touch-sensitive display 112 issometimes called a “touch screen” for convenience, and is sometimesknown as or called a touch-sensitive display system. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPU's) 120, peripherals interface 118, RF circuitry 108, audiocircuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem106, other input or control devices 116, and external port 124. Device100 optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensity of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100optionally includes one or more tactile output generators 167 forgenerating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact) on the touchsensitive surface, or to a substitute (proxy) for the force or pressureof a contact on the touch sensitive surface. The intensity of a contacthas a range of values that includes at least four distinct values andmore typically includes hundreds of distinct values (e.g., at least256). Intensity of a contact is, optionally, determined (or measured)using various approaches and various sensors or combinations of sensors.For example, one or more force sensors underneath or adjacent to thetouch-sensitive surface are, optionally, used to measure force atvarious points on the touch-sensitive surface. In some implementations,force measurements from multiple force sensors are combined (e.g., aweighted average) to determine an estimated force of a contact.Similarly, a pressure-sensitive tip of a stylus is, optionally, used todetermine a pressure of the stylus on the touch-sensitive surface.Alternatively, the size of the contact area detected on thetouch-sensitive surface and/or changes thereto, the capacitance of thetouch-sensitive surface proximate to the contact and/or changes thereto,and/or the resistance of the touch-sensitive surface proximate to thecontact and/or changes thereto are, optionally, used as a substitute forthe force or pressure of the contact on the touch-sensitive surface. Insome implementations, the substitute measurements for contact force orpressure are used directly to determine whether an intensity thresholdhas been exceeded (e.g., the intensity threshold is described in unitscorresponding to the substitute measurements). In some implementations,the substitute measurements for contact force or pressure are convertedto an estimated force or pressure and the estimated force or pressure isused to determine whether an intensity threshold has been exceeded(e.g., the intensity threshold is a pressure threshold measured in unitsof pressure).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU 120 and the peripherals interface118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), voice over InternetProtocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet messageaccess protocol (IMAP) and/or post office protocol (POP)), instantmessaging (e.g., extensible messaging and presence protocol (XMPP),Session Initiation Protocol for Instant Messaging and PresenceLeveraging Extensions (SIMPLE), Instant Messaging and Presence Service(IMPS)), and/or Short Message Service (SMS), or any other suitablecommunication protocol, including communication protocols not yetdeveloped as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch screen 112 and other input control devices 116, to peripheralsinterface 118. I/O subsystem 106 optionally includes display controller156, optical sensor controller 158, intensity sensor controller 159,haptic feedback controller 161 and one or more input controllers 160 forother input or control devices. The one or more input controllers 160receive/send electrical signals from/to other input or control devices116. The other input control devices 116 optionally include physicalbuttons (e.g., push buttons, rocker buttons, etc.), dials, sliderswitches, joysticks, click wheels, and so forth. In some alternateembodiments, input controller(s) 160 are, optionally, coupled to any (ornone) of the following: a keyboard, infrared port, USB port, and apointer device such as a mouse. The one or more buttons (e.g., 208, FIG.2) optionally include an up/down button for volume control of speaker111 and/or microphone 113. The one or more buttons optionally include apush button (e.g., 206, FIG. 2).

Touch-sensitive display 112 provides an input interface and an outputinterface between the device and a user. Display controller 156 receivesand/or sends electrical signals from/to touch screen 112. Touch screen112 displays visual output to the user. The visual output optionallyincludes graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output corresponds to user-interface objects.

Touch screen 112 has a touch-sensitive surface, sensor or set of sensorsthat accepts input from the user based on haptic and/or tactile contact.Touch screen 112 and display controller 156 (along with any associatedmodules and/or sets of instructions in memory 102) detect contact (andany movement or breaking of the contact) on touch screen 112 andconverts the detected contact into interaction with user-interfaceobjects (e.g., one or more soft keys, icons, web pages or images) thatare displayed on touch screen 112. In an exemplary embodiment, a pointof contact between touch screen 112 and the user corresponds to a fingerof the user.

Touch screen 112 optionally uses LCD (liquid crystal display)technology, LPD (light emitting polymer display) technology, or LED(light emitting diode) technology, although other display technologiesare used in other embodiments. Touch screen 112 and display controller156 optionally detect contact and any movement or breaking thereof usingany of a plurality of touch sensing technologies now known or laterdeveloped, including but not limited to capacitive, resistive, infrared,and surface acoustic wave technologies, as well as other proximitysensor arrays or other elements for determining one or more points ofcontact with touch screen 112. In an exemplary embodiment, projectedmutual capacitance sensing technology is used, such as that found in theiPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, Calif.

Touch screen 112 optionally has a video resolution in excess of 100 dpi.In some embodiments, the touch screen has a video resolution ofapproximately 160 dpi. The user optionally makes contact with touchscreen 112 using any suitable object or appendage, such as a stylus, afinger, and so forth. In some embodiments, the user interface isdesigned to work primarily with finger-based contacts and gestures,which can be less precise than stylus-based input due to the larger areaof contact of a finger on the touch screen. In some embodiments, thedevice translates the rough finger-based input into a precisepointer/cursor position or command for performing the actions desired bythe user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch screen 112 or anextension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled to optical sensor controller 158in I/O subsystem 106. Optical sensor 164 optionally includescharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor 164 receives light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor 164 optionally capturesstill images or video. In some embodiments, an optical sensor is locatedon the back of device 100, opposite touch screen display 112 on thefront of the device, so that the touch screen display is enabled for useas a viewfinder for still and/or video image acquisition. In someembodiments, another optical sensor is located on the front of thedevice so that the user's image is, optionally, obtained forvideoconferencing while the user views the other video conferenceparticipants on the touch screen display.

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled tointensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor 165 optionally includes one or more piezoresistive strain gauges,capacitive force sensors, electric force sensors, piezoelectric forcesensors, optical force sensors, capacitive touch-sensitive surfaces, orother intensity sensors (e.g., sensors used to measure the force (orpressure) of a contact on a touch-sensitive surface). Contact intensitysensor 165 receives contact intensity information (e.g., pressureinformation or a proxy for pressure information) from the environment.In some embodiments, at least one contact intensity sensor is collocatedwith, or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 112). In some embodiments, at least one contact intensitysensor is located on the back of device 100, opposite touch screendisplay 112 which is located on the front of device 100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled to peripherals interface 118.Alternately, proximity sensor 166 is coupled to input controller 160 inI/O subsystem 106. In some embodiments, the proximity sensor turns offand disables touch screen 112 when the multifunction device is placednear the user's ear (e.g., when the user is making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled tohaptic feedback controller 161 in I/O subsystem 106. Tactile outputgenerator 167 optionally includes one or more electroacoustic devicessuch as speakers or other audio components and/or electromechanicaldevices that convert energy into linear motion such as a motor,solenoid, electroactive polymer, piezoelectric actuator, electrostaticactuator, or other tactile output generating component (e.g., acomponent that converts electrical signals into tactile outputs on thedevice). Contact intensity sensor 165 receives tactile feedbackgeneration instructions from haptic feedback module 133 and generatestactile outputs on device 100 that are capable of being sensed by a userof device 100. In some embodiments, at least one tactile outputgenerator is collocated with, or proximate to, a touch-sensitive surface(e.g., touch-sensitive display system 112) and, optionally, generates atactile output by moving the touch-sensitive surface vertically (e.g.,in/out of a surface of device 100) or laterally (e.g., back and forth inthe same plane as a surface of device 100). In some embodiments, atleast one tactile output generator sensor is located on the back ofdevice 100, opposite touch screen display 112 which is located on thefront of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled to peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled to an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, text input module (or setof instructions) 134, Global Positioning System (GPS) module (or set ofinstructions) 135, and applications (or sets of instructions) 136.Furthermore, in some embodiments memory 102 stores device/globalinternal state 157, as shown in FIGS. 1A and 3. Device/global internalstate 157 includes one or more of: active application state, indicatingwhich applications, if any, are currently active; display state,indicating what applications, views or other information occupy variousregions of touch screen display 112; sensor state, including informationobtained from the device's various sensors and input control devices116; and location information concerning the device's location and/orattitude.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VxWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used on iPod (trademark of Apple Inc.) devices.

Contact/motion module 130 optionally detects contact with touch screen112 (in conjunction with display controller 156) and other touchsensitive devices (e.g., a touchpad or physical click wheel).Contact/motion module 130 includes various software components forperforming various operations related to detection of contact, such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact) determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts) or to multiple simultaneous contacts (e.g.,“multitouch”/multiple finger contacts). In some embodiments,contact/motion module 130 and display controller 156 detect contact on atouchpad.

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns and intensities. Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event.

Graphics module 132 includes various known software components forrendering and displaying graphics on touch screen 112 or other display,including components for changing the visual impact (e.g., brightness,transparency, saturation, contrast or other visual property) of graphicsthat are displayed. As used herein, the term “graphics” includes anyobject that can be displayed to a user, including without limitationtext, web pages, icons (such as user-interface objects including softkeys), digital images, videos, animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions used by tactile output generator(s) 167 toproduce tactile outputs at one or more locations on device 100 inresponse to user interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, and text input module 134, contactsmodule 137 are, optionally, used to manage an address book or contactlist (e.g., stored in application internal state 192 of contacts module137 in memory 102 or memory 370), including: adding name(s) to theaddress book; deleting name(s) from the address book; associatingtelephone number(s), e-mail address(es), physical address(es) or otherinformation with a name; associating an image with a name; categorizingand sorting names; providing telephone numbers or e-mail addresses toinitiate and/or facilitate communications by telephone 138, videoconference 139, e-mail 140, or IM 141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, contact module130, graphics module 132, and text input module 134, telephone module138 are, optionally, used to enter a sequence of characterscorresponding to a telephone number, access one or more telephonenumbers in address book 137, modify a telephone number that has beenentered, dial a respective telephone number, conduct a conversation anddisconnect or hang up when the conversation is completed. As notedabove, the wireless communication optionally uses any of a plurality ofcommunications standards, protocols and technologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, optical sensor164, optical sensor controller 158, contact module 130, graphics module132, text input module 134, contact list 137, and telephone module 138,videoconferencing module 139 includes executable instructions toinitiate, conduct, and terminate a video conference between a user andone or more other participants in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, e-mail client module 140 includes executable instructions tocreate, send, receive, and manage e-mail in response to userinstructions. In conjunction with image management module 144, e-mailclient module 140 makes it very easy to create and send e-mails withstill or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, the instant messaging module 141 includes executableinstructions to enter a sequence of characters corresponding to aninstant message, to modify previously entered characters, to transmit arespective instant message (for example, using a Short Message Service(SMS) or Multimedia Message Service (MMS) protocol for telephony-basedinstant messages or using XMPP, SIMPLE, or IMPS for Internet-basedinstant messages), to receive instant messages and to view receivedinstant messages. In some embodiments, transmitted and/or receivedinstant messages optionally include graphics, photos, audio files, videofiles and/or other attachments as are supported in a MMS and/or anEnhanced Messaging Service (EMS). As used herein, “instant messaging”refers to both telephony-based messages (e.g., messages sent using SMSor MMS) and Internet-based messages (e.g., messages sent using XMPP,SIMPLE, or IMPS).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, GPS module 135, map module 154, and music player module 146,workout support module 142 includes executable instructions to createworkouts (e.g., with time, distance, and/or calorie burning goals);communicate with workout sensors (sports devices); receive workoutsensor data; calibrate sensors used to monitor a workout; select andplay music for a workout; and display, store and transmit workout data.

In conjunction with touch screen 112, display controller 156, opticalsensor(s) 164, optical sensor controller 158, contact module 130,graphics module 132, and image management module 144, camera module 143includes executable instructions to capture still images or video(including a video stream) and store them into memory 102, modifycharacteristics of a still image or video, or delete a still image orvideo from memory 102.

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, text input module 134, and cameramodule 143, image management module 144 includes executable instructionsto arrange, modify (e.g., edit), or otherwise manipulate, label, delete,present (e.g., in a digital slide show or album), and store still and/orvideo images.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, browser module 147 includes executable instructions tobrowse the Internet in accordance with user instructions, includingsearching, linking to, receiving, and displaying web pages or portionsthereof, as well as attachments and other files linked to web pages.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, e-mail client module 140, and browser module 147, calendarmodule 148 includes executable instructions to create, display, modify,and store calendars and data associated with calendars (e.g., calendarentries, to do lists, etc.) in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, and browser module 147, widget modules 149 aremini-applications that are, optionally, downloaded and used by a user(e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, and browser module 147, the widget creator module 150 are,optionally, used by a user to create widgets (e.g., turning auser-specified portion of a web page into a widget).

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, and text input module 134,search module 151 includes executable instructions to search for text,music, sound, image, video, and/or other files in memory 102 that matchone or more search criteria (e.g., one or more user-specified searchterms) in accordance with user instructions.

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, audio circuitry 110, speaker111, RF circuitry 108, and browser module 147, video and music playermodule 152 includes executable instructions that allow the user todownload and play back recorded music and other sound files stored inone or more file formats, such as MP3 or AAC files, and executableinstructions to display, present or otherwise play back videos (e.g., ontouch screen 112 or on an external, connected display via external port124). In some embodiments, device 100 optionally includes thefunctionality of an MP3 player, such as an iPod (trademark of AppleInc.).

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, and text input module 134, notes module153 includes executable instructions to create and manage notes, to dolists, and the like in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, GPS module 135, and browser module 147, map module 154 are,optionally, used to receive, display, modify, and store maps and dataassociated with maps (e.g., driving directions; data on stores and otherpoints of interest at or near a particular location; and otherlocation-based data) in accordance with user instructions.

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, audio circuitry 110, speaker111, RF circuitry 108, text input module 134, e-mail client module 140,and browser module 147, online video module 155 includes instructionsthat allow the user to access, browse, receive (e.g., by streamingand/or download), play back (e.g., on the touch screen or on anexternal, connected display via external port 124), send an e-mail witha link to a particular online video, and otherwise manage online videosin one or more file formats, such as H.264. In some embodiments, instantmessaging module 141, rather than e-mail client module 140, is used tosend a link to a particular online video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 137-151, 155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch sensitivedisplay 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display 112, as part of a multi-touchgesture). Peripherals interface 118 transmits information it receivesfrom I/O subsystem 106 or a sensor, such as proximity sensor 166,accelerometer(s) 168, and/or microphone 113 (through audio circuitry110). Information that peripherals interface 118 receives from I/Osubsystem 106 includes information from touch-sensitive display 112 or atouch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch sensitive display 112 displays more than one view. Views aremade up of controls and other elements that a user can see on thedisplay.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay 112, and lift-off of the touch (touch end). In some embodiments,the event also includes information for one or more associated eventhandlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display 112, when a touch is detected on touch-sensitivedisplay 112, event comparator 184 performs a hit test to determine whichof the three user-interface objects is associated with the touch(sub-event). If each displayed object is associated with a respectiveevent handler 190, the event comparator uses the result of the hit testto determine which event handler 190 should be activated. For example,event comparator 184 selects an event handler associated with thesub-event and the object triggering the hit test.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoplayer module 145. In some embodiments, object updater 177 creates andupdates objects used in application 136-1. For example, object updater177 creates a new user-interface object or updates the position of auser-interface object. GUI updater 178 updates the GUI. For example, GUIupdater 178 prepares display information and sends it to graphics module132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen 112 in accordance with some embodiments. The touch screenoptionally displays one or more graphics within user interface (UI) 200.In this embodiment, as well as others described below, a user is enabledto select one or more of the graphics by making a gesture on thegraphics, for example, with one or more fingers 202 (not drawn to scalein the figure) or one or more styluses 203 (not drawn to scale in thefigure). In some embodiments, selection of one or more graphics occurswhen the user breaks contact with the one or more graphics. In someembodiments, the gesture optionally includes one or more taps, one ormore swipes (from left to right, right to left, upward and/or downward)and/or a rolling of a finger (from right to left, left to right, upwardand/or downward) that has made contact with device 100. In someimplementations or circumstances, inadvertent contact with a graphicdoes not select the graphic. For example, a swipe gesture that sweepsover an application icon optionally does not select the correspondingapplication when the gesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on touch screen 112.

In one embodiment, device 100 includes touch screen 112, menu button204, push button 206 for powering the device on/off and locking thedevice, volume adjustment button(s) 208, Subscriber Identity Module(SIM) card slot 210, head set jack 212, and docking/charging externalport 124. Push button 206 is, optionally, used to turn the power on/offon the device by depressing the button and holding the button in thedepressed state for a predefined time interval; to lock the device bydepressing the button and releasing the button before the predefinedtime interval has elapsed; and/or to unlock the device or initiate anunlock process. In an alternative embodiment, device 100 also acceptsverbal input for activation or deactivation of some functions throughmicrophone 113. Device 100 also, optionally, includes one or morecontact intensity sensors 165 for detecting intensity of contacts ontouch screen 112 and/or one or more tactile output generators 167 forgenerating tactile outputs for a user of device 100.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3 are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces (“UI”)that is, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on portable multifunction device 100 in accordance withsome embodiments. Similar user interfaces are, optionally, implementedon device 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) 402 for wireless communication(s),        such as cellular and Wi-Fi signals;    -   Time 404;    -   Bluetooth indicator 405;    -   Battery status indicator 406;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser;” and        -   Icon 422 for video and music player module 152, also            referred to as iPod (trademark of Apple Inc.) module 152,            labeled “iPod;” and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Text;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online Video”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Map;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, which            provides access to settings for device 100 and its various            applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely exemplary. For example, icon 422 for video and music playermodule 152 are labeled “Music” or “Music Player.” Other labels are,optionally, used for various application icons. In some embodiments, alabel for a respective application icon includes a name of anapplication corresponding to the respective application icon. In someembodiments, a label for a particular application icon is distinct froma name of an application corresponding to the particular applicationicon.

FIG. 4B illustrates an exemplary user interface on a device (e.g.,device 300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tabletor touchpad 355, FIG. 3) that is separate from the display 450 (e.g.,touch screen display 112). Device 300 also, optionally, includes one ormore contact intensity sensors (e.g., one or more of sensors 357) fordetecting intensity of contacts on touch-sensitive surface 451 and/orone or more tactile output generators 359 for generating tactile outputsfor a user of device 300.

Although some of the examples which follow will be given with referenceto inputs on touch screen display 112 (where the touch sensitive surfaceand the display are combined), in some embodiments, the device detectsinputs on a touch-sensitive surface that is separate from the display,as shown in FIG. 4B. In some embodiments the touch sensitive surface(e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) thatcorresponds to a primary axis (e.g., 453 in FIG. 4B) on the display(e.g., 450). In accordance with these embodiments, the device detectscontacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface451 at locations that correspond to respective locations on the display(e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470).In this way, user inputs (e.g., contacts 460 and 462, and movementsthereof) detected by the device on the touch-sensitive surface (e.g.,451 in FIG. 4B) are used by the device to manipulate the user interfaceon the display (e.g., 450 in FIG. 4B) of the multifunction device whenthe touch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse based input or stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B)while the cursor is over a particular user interface element (e.g., abutton, window, slider or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations that include a touch-screen display(e.g., touch-sensitive display system 112 in FIG. 1A or touch screen 112in FIG. 4A) that enables direct interaction with user interface elementson the touch-screen display, a detected contact on the touch-screen actsas a “focus selector,” so that when an input (e.g., a press input by thecontact) is detected on the touch-screen display at a location of aparticular user interface element (e.g., a button, window, slider orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementationsfocus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch-screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch-screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

The user interface figures described below include various intensitydiagrams that show the current intensity of the contact on thetouch-sensitive surface relative to one or more intensity thresholds(e.g., a contact detection intensity threshold IT₀, a light pressintensity threshold IT_(L), a deep press intensity threshold IT_(D),and/or one or more other intensity thresholds). This intensity diagramis typically not part of the displayed user interface, but is providedto aid in the interpretation of the figures. In some embodiments, thelight press intensity threshold corresponds to an intensity at which thedevice will perform operations typically associated with clicking abutton of a physical mouse or a trackpad. In some embodiments, the deeppress intensity threshold corresponds to an intensity at which thedevice will perform operations that are different from operationstypically associated with clicking a button of a physical mouse or atrackpad. In some embodiments, when a contact is detected with anintensity below the light press intensity threshold (e.g., and above anominal contact-detection intensity threshold IT₀ below which thecontact is no longer detected), the device will move a focus selector inaccordance with movement of the contact on the touch-sensitive surfacewithout performing an operation associated with the light pressintensity threshold or the deep press intensity threshold. Generally,unless otherwise stated, these intensity thresholds are consistentbetween different sets of user interface figures.

An increase of intensity of the contact from an intensity below thelight press intensity threshold IT_(L) to an intensity between the lightpress intensity threshold IT_(L) and the deep press intensity thresholdIT_(D) is sometimes referred to as a “light press” input. An increase ofintensity of the contact from an intensity below the deep pressintensity threshold IT_(D) to an intensity above the deep pressintensity threshold IT_(D) is sometimes referred to as a “deep press”input. An increase of intensity of the contact from an intensity belowthe contact-detection intensity threshold IT₀ to an intensity betweenthe contact-detection intensity threshold IT₀ and the light pressintensity threshold IT_(L) is sometimes referred to as detecting thecontact on the touch-surface. A decrease of intensity of the contactfrom an intensity above the contact-detection intensity threshold IT₀ toan intensity below the contact intensity threshold IT₀ is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments IT₀ is zero. In some embodiments IT₀ is greater thanzero. In some illustrations a shaded circle or oval is used to representintensity of a contact on the touch-sensitive surface. In someillustrations a circle or oval without shading is used represent arespective contact on the touch-sensitive surface without specifying theintensity of the respective contact.

In some embodiments described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., a “down stroke” of the respective pressinput). In some embodiments, the press input includes an increase inintensity of the respective contact above the press-input intensitythreshold and a subsequent decrease in intensity of the contact belowthe press-input intensity threshold, and the respective operation isperformed in response to detecting the subsequent decrease in intensityof the respective contact below the press-input threshold (e.g., an “upstroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90% or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., an “up stroke” of therespective press input). Similarly, in some embodiments, the press inputis detected only when the device detects an increase in intensity of thecontact from an intensity at or below the hysteresis intensity thresholdto an intensity at or above the press-input intensity threshold and,optionally, a subsequent decrease in intensity of the contact to anintensity at or below the hysteresis intensity, and the respectiveoperation is performed in response to detecting the press input (e.g.,the increase in intensity of the contact or the decrease in intensity ofthe contact, depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting either: an increase inintensity of a contact above the press-input intensity threshold, anincrease in intensity of a contact from an intensity below thehysteresis intensity threshold to an intensity above the press-inputintensity threshold, a decrease in intensity of the contact below thepress-input intensity threshold, and/or a decrease in intensity of thecontact below the hysteresis intensity threshold corresponding to thepress-input intensity threshold. Additionally, in examples where anoperation is described as being performed in response to detecting adecrease in intensity of a contact below the press-input intensitythreshold, the operation is, optionally, performed in response todetecting a decrease in intensity of the contact below a hysteresisintensity threshold corresponding to, and lower than, the press-inputintensity threshold.

User Interfaces and Associated Processes Providing Tactile Feedback forOperations Performed on in a User Interface

Many electronic devices have graphical user interfaces that include userinterface objects. There are usually many operations which can beperformed on the interface objects. For example, an interface object canbe snapped to a guideline or removed from a guideline. Another examplewould be moving a user interface object (e.g., a file) into or out of afolder. There is often a need to provide efficient and convenient waysfor users to receive feedback for operations performed on these userinterface objects. The embodiments below improve on existing methods bygenerating tactile outputs for the user corresponding to the operationsperformed.

FIGS. 5A-5O illustrate exemplary user interfaces for providing tactilefeedback for operations performed in a user interface in accordance withsome embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processesdescribed below with reference to FIGS. 6A-6C.

FIG. 5A illustrates an example of a user interface that includes a userinterface object. User interface 10200 in FIGS. 5A-5C is displayed ondisplay 450 of a device (e.g., device 300) and is responsive to contacts(e.g., a finger contact) on touch-sensitive surface 451. User interface10200 includes user interface object 10202 and, per some embodiments,object placement guide 10208. FIG. 5A further illustrates contact 10204at position 10204-a on touch-sensitive surface 451 and a displayedrepresentation of a focus selector (e.g., cursor 10206) corresponding tocontact 10204.

In some embodiments, the device is an electronic device with a separatedisplay (e.g., display 450) and a separate touch-sensitive surface(e.g., touch-sensitive surface 451). In some embodiments, the device isportable multifunction device 100, the display is touch-sensitivedisplay system 112, and the touch-sensitive surface includes tactileoutput generators 167 on the display (FIG. 1A). For convenience ofexplanation, the embodiments described with reference to FIGS. 5A-5O andFIGS. 6A-6C will be discussed with reference to display 450 and aseparate touch-sensitive surface 451, however analogous operations are,optionally, performed on a device with a touch-sensitive display system112 in response to detecting the contacts described in FIGS. 5A-5O onthe touch-sensitive display system 112 while displaying the userinterfaces shown in FIGS. 5A-5O on the touch-sensitive display system112; in such embodiments, the focus selector is, optionally: arespective contact, a representative point corresponding to a contact(e.g., a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112, in place of cursor 10206.

FIGS. 5A-5B illustrate an example of performing an operation thatincludes snapping a user interface object into a respective objectplacement guide. In this example, the device detects contact 10204 andmovement 10210 of contact 10204 from position 10204-a in FIG. 5A toposition 10204-b in FIG. 5B on touch-sensitive surface 451. In responseto detecting movement 10210, which corresponds to moving user interfaceobject 10206 within a snapping distance of object placement guide 10208,the device snaps user interface object 10202 into object placement guide10208 and generates tactile output 10211 on touch-sensitive surface 451.

FIGS. 5B-5C illustrate an example of reversing the operation shown inFIGS. 5B-5C by snapping a user interface object out of a respectiveobject placement guide. In this example, the device detects contact10204 and movement 10212 of contact 10204 from position 10204-b in FIG.5B to position 10204-c in FIG. 5C on touch-sensitive surface 451. Inresponse to detecting movement 10212, which corresponds to moving userinterface object 10206 at or beyond an unsnapping distance of objectplacement guide 10208, the device snaps user interface object 10202 outof object placement guide 10208 (e.g., reversing the operation describedabove with reference to FIGS. 5A-5B). The device also generates tactileoutput 10213 on touch-sensitive surface 451 in response to movement10212 of contact 10204 that corresponds to snapping user interfaceobject 10202 out of object placement guide 10208.

In some embodiments, the device compares a respective amount of timebetween movement 10210 in FIG. 5B and movement 10212 in FIG. 5C with apredefined time threshold (e.g., the device determines a magnitude of apause time between the end of the movement 10210 and the beginning ofmovement 10212 and compares the magnitude of the pause time with thepredefined time threshold). In some embodiments, in accordance with adetermination that the respective amount of time (e.g., the pause time)between movement 10210 and movement 10212 is less than the predefinedtime threshold, the device forgoes generating tactile output 10211and/or tactile output 10213. In contrast, in some embodiments, inaccordance with a determination that the respective amount of time(e.g., the pause time) between movement 10210 and movement 10212 isgreater than the predefined time threshold, the device generates tactileoutput 10211 and tactile output 10213, as described in greater detailabove. In some embodiments, when the respective amount of time is lessthan the predefined time threshold, the tactile output corresponding toperforming the operation (e.g., tactile output 10211) is generated andthe tactile output corresponding to reversing the operation (e.g.,tactile output 10213) is not generated. In some embodiments, when therespective amount of time is less than the predefined time threshold,the tactile output corresponding to performing the operation (e.g.,tactile output 10211) and the tactile output corresponding to reversingthe operation (e.g., tactile output 10213) are both not generated. Insome embodiments, when the respective amount of time is less than thepredefined time threshold, the tactile output corresponding toperforming the operation (e.g., tactile output 10211) is not generatedand the tactile output corresponding to reversing the operation (e.g.,tactile output 10213) is generated. Forgoing generating one or moretactile outputs corresponding to performing and reversing the operationwhen the operation is performed and reversed quickly (e.g., before thepredefined time threshold has elapsed) prevents tactile outputs frombeing generated when the user accidentally performs and then reversesthe operation. These tactile outputs, would likely, if generated,confuse the user or distract the user from other more important tactileand visual feedback. As such, selectively suppressing (e.g., forgoing)generating tactile outputs based on one or more predefined timethresholds, as described above, provides a more efficient and intuitiveuser interface, thereby improving the user experience when interactingwith the user interface.

FIG. 5D illustrates another example of a user interface that includes auser interface object. In this example, user interface 10200 in FIGS.5D-5F includes user interface object 10222 and folder 10228 which is atrash folder used to mark data for deletion. FIG. 5D further illustratescontact 10224 at position 10224-a on touch-sensitive surface 451 and adisplayed representation of a focus selector (e.g., cursor 10206)corresponding to contact 10224.

FIGS. 5D-5E illustrate an example of performing an operation. In thisexample, performing an operation includes marking data corresponding toa user interface object for deletion. In this example, the devicedetects contact 10224 and movement 10230 of contact 10224 from position10224-a in FIG. 5D to position 10224-b in FIG. 5E on touch-sensitivesurface 451. In response to detecting movement 10230, the device movesuser interface object 10222 over folder 10228, as shown in FIG. 5E andthe device marks data corresponding to user interface object 10222 fordeletion and generates tactile output 10231 on touch-sensitive surface451. Optionally, marking the user interface object for deletion isperformed in response to detecting an input such as liftoff of thecontact.

FIGS. 5E-5F illustrate an example of reversing the operation shown inFIGS. 5D-5E, by unmarking data corresponding to a user interface objectfor deletion. In this example, the device detects contact 10224 andmovement 10232 of contact 10224 from position 10224-b in FIG. 5E toposition 10224-c in FIG. 5F on touch-sensitive surface 451. In responseto detecting movement 10232, the device moves user interface object 102away from folder icon 10228 and unmarks data corresponding to userinterface object 10202 for deletion (e.g., reversing the data markingoperation described above with reference to FIGS. 5D-5E). The devicealso generates tactile output 10233 on touch-sensitive surface 451 inresponse to movement 10232 of contact 10224 that corresponds tounmarking data user interface object 10222 for deletion. Incircumstances where marking the user interface object for deletion isperformed in response to detecting an input such as liftoff of thecontact, contact 10224 in FIG. 5F is optionally a different contact fromcontact 10224 in FIG. 5E.

FIG. 5G illustrates another example of a user interface that includes auser interface object. In this example, user interface 10200 in FIGS.5G-5I includes user interface object 10242, representing a file in thisexample, and folder 10248 representing a directory in a file system.FIG. 5G further illustrates contact 10244 at position 10244-a ontouch-sensitive surface 451 and a displayed representation of a focusselector (e.g., cursor 10206) corresponding to contact 10244.

FIGS. 5G-5H illustrate another example of performing an operation. Inthis example the operation includes placing a file in a directory. Inthis example, the device detects contact 10244 and movement 10250 ofcontact 10244 from position 10244-a in FIG. 5G to position 10244-b inFIG. 5H on touch-sensitive surface 451. In response to detectingmovement 10250, the device moves user interface object 10242 over folder10248 and the device places the file, represented by user interfaceobject 10242, in the directory, represented by folder 10248 andgenerates tactile output 10251 on touch-sensitive surface 451.Optionally, placing the file represented by the user interface object inthe folder is performed in response to detecting an input such asliftoff of the contact.

FIGS. 5H-5I illustrate an example of reversing the operation shown inFIGS. 5G-5H by removing a file from a directory. In this example, thedevice detects contact 10244 and movement 10252 of contact 10244 fromposition 10244-b in FIG. 5H to position 10244-c in FIG. 5I ontouch-sensitive surface 451. In response to detecting movement 10252,the device moves user interface object 10242 away from folder icon 10248and removes the file represented by user interface object 10242 from thedirectory represented by folder 10248 (e.g., reversing the operationdescribed above with reference to FIGS. 5G-5H). The device alsogenerates tactile output 10253 on touch-sensitive surface 451 inresponse movement 10252 of contact 10244 that corresponds to removingthe file from the directory. In circumstances where placing the filerepresented by the user interface object in the folder is performed inresponse to detecting an input such as liftoff of the contact, contact10244 in FIG. 5I is optionally a different contact from contact 10244 inFIG. 5H.

FIG. 5J illustrates another example of a user interface that includes auser interface object. In this example, user interface 10200 in FIGS.5J-5L includes user interface object 10262, corresponding to anapplication (or an application launch icon) and application launchregion 10268. FIG. 5J further illustrates contact 10264 at position10264-a on touch-sensitive surface 451 and a displayed representation ofa focus selector (e.g., cursor 10206) corresponding to contact 10264.

FIGS. 5J-5K illustrate an example of performing an operation. In thisexample the operation includes placing a user interface object in anapplication launch region. In this example, the device detects contact10264 and movement 10270 of contact 10264 from position 10264-a in FIG.5J to position 10264-b in FIG. 5K on touch-sensitive surface 451. Inresponse to detecting movement 10270, the device moves user interfaceobject 10262 over application launch region 10268 and places userinterface object 10262 in application launch region 10268 and generatestactile output 10271 on touch-sensitive surface 451. Optionally, placingthe user interface object in the application launch region is performedin response to detecting an input such as liftoff of the contact.

FIGS. 5K-5L illustrate an example of reversing the operation shown inFIGS. 5J-5K by removing a user interface object from an applicationlaunch region. In this example, the device detects contact 10264 andmovement 10272 of contact 10264 from position 10264-b in FIG. 5K toposition 10264-c in FIG. 5L on touch-sensitive surface 451. In responseto detecting movement 10272, the device moves user interface object10262 away from application launch region 10268 and removes userinterface object 10262 from application launch region 10268 (e.g.,reversing the operation described above with reference to FIGS. 5J-5K).The device also generates tactile output 10273 on touch-sensitivesurface 451 in response to movement 10272 of contact 10264 thatcorresponds to removing user interface object 10262 from applicationlaunch region 10268. In circumstances where placing the user interfaceobject in the application launch region is performed in response todetecting an input such as liftoff of the contact, contact 10264 in FIG.5L is optionally a different contact from contact 10264 in FIG. 5K.

FIGS. 5M-5O illustrate example waveforms of movement profiles forgenerating the tactile output. FIG. 5M illustrates a triangle waveformwith period 10280-1. FIG. 5N illustrates a square waveform with period10280-2 and FIG. 5O illustrates a sawtooth waveform with period 10280-3.One of these movement profiles illustrated in FIGS. 5M-5O is,optionally, be utilized when generating a tactile output correspondingperforming an operation (e.g., tactile outputs 10211, 10231, 10251 or10271) or reversing a performed operation (e.g., tactile outputs 10213,10233, 10253 or 10273), as discussed above. In some embodiments, anotherwaveform is used to generate tactile outputs corresponding to thedifferent operations described with reference to FIGS. 5A-5L, above. Insome embodiments the tactile outputs corresponding to performing anoperation (e.g., tactile outputs 10211, 10231, 10251 or 10271) aregenerated using the same waveform. In some embodiments the tactileoutputs corresponding to reversing an operation (e.g., tactile outputs10213, 10233, 10253 or 10273) are generated using the same waveform. Insome embodiments the tactile outputs corresponding to performing anoperation (e.g., tactile outputs 10211, 10231, 10251 or 10271) aregenerated using a first waveform that is different from a secondwaveform used to generate the tactile outputs corresponding to reversingan operation (e.g., tactile outputs 10213, 10233, 10253 or 10273).

FIGS. 6A-6C are flow diagrams illustrating a method 10300 of providingtactile feedback for operations performed in a user interface inaccordance with some embodiments. Method 10300 is performed at anelectronic device (e.g., device 300, FIG. 3, or portable multifunctiondevice 100, FIG. 1A) with a display and a touch-sensitive surface. Insome embodiments, the display is a touch screen display and thetouch-sensitive surface is on the display. In some embodiments, thedisplay is separate from the touch-sensitive surface. Some operations inmethod 10300 are, optionally, combined and/or the order of someoperations is, optionally, changed.

As described below, the method 10300 provides an intuitive way toprovide tactile feedback for operations performed in a user interface.The method reduces the cognitive burden on a user when performingoperations in a user interface, thereby creating a more efficienthuman-machine interface. For battery-operated electronic devices,enabling a user to perform operations in a user interface faster andmore efficiently conserves power and increases the time between batterycharges.

The device displays (10302) a user interface object. FIG. 5A, forexample, shows user interface object 10202, displayed in graphical userinterface 10200. The device detects (10304) a contact (e.g., a fingercontact) on the touch-sensitive surface. For example, FIG. 5A showscontact 10204 at position 10204-a on touch-sensitive surface 451.

The device detects (10306) a first movement of the contact across thetouch-sensitive surface, where the first movement corresponds toperforming an operation on the user interface object. For example, FIG.5B shows contact 10204 and subsequent movement 10210 on touch-sensitivesurface 451 corresponding to snapping user interface object 10202 intoobject placement guide 10208. In response to detecting (10310) the firstmovement, the device performs (10312) the operation and generates(10314) a first tactile output on the touch-sensitive surface. FIG. 5B,for example, shows user interface object 10202 snapping into objectplacement guide 10208 and tactile output 10211 generated ontouch-sensitive surface 451.

The device detects (10316) a second movement of the contact across thetouch-sensitive surface, where the second movement corresponds toreversing the operation on the user interface object. For example, FIG.5C shows contact 10204 and subsequent movement 10212 on touch-sensitivesurface corresponding to snapping user interface object 10202 out ofobject placement guide 10208.

In some embodiments or circumstances, the first movement of the contactacross the touch-sensitive surface and the second movement of thecontact across the touch-sensitive surface are (10318) part of a singlecontinuous gesture performed without detecting a liftoff of the contactfrom the touch-sensitive surface. In some embodiments, even if there isa pause in movement of the contact, the first movement and the secondmovement are considered to be part of the same continuous gesture aslong as the contact continues to be detected on the touch-sensitivesurface. In some embodiments, if the same first and second movement aredetected as part of two different gestures (e.g., there is a liftoff ofthe contact between when the first movement is detected and when thesecond movement is detected), then the same tactile output is generatedin response to both performing the operation and reversing theoperation). In some embodiments, if the same first and second movementare detected as part of two different gestures (e.g., there is a liftoffof the contact between when the first movement is detected and when thesecond movement is detected) then different tactile outputs are stillgenerated in response to both performing the operation and reversing theoperation. For example, FIGS. 5A-5C show contact 10204 moving fromposition 10204-a to position 10204-b (shown in FIGS. 5A-5B) whichcorresponds to snapping user interface object 10202 to object placementguide 10208, then moving from position 10204-b to position 10204-c(shown in FIGS. 5B-5C) which corresponds to unsnapping user interfaceobject 10202 from object placement guide 10208. In this example, tactileoutput 10211 is generated in response to contact 10204 moving fromposition 10204-a to position 10204-b and tactile output 10213 isgenerated in response to contact 10204 moving from position 10204-b toposition 10204-c. In some embodiments, tactile output 10213 (sometimescalled the second tactile output) is generated in response to detectinga movement corresponding to reversing a prior operation on a respectiveuser interface object. In some embodiments, tactile output 10211(sometimes called the first tactile output) is generated in response todetecting a movement corresponding to perform an operation that is not areversal of a prior operation (e.g., an immediately prior operation) ona respective user interface object.

In response to detecting (10320) the second movement, the devicereverses (10322) the operation. It should be understood that reversingan operation does not necessarily entail performing an exact mirrorimage of the procedure undertaken to perform the operation. For example,to snap an object to a guide the object is moved to a position within asnapping distance from the guide, while to move an object away from theguide, movement of a contact is detected that corresponds to movement ofthe object more than an unsnapping distance from the guide, withoutrespect to the particular path taken by the contact or the object. FIG.5C, for example, shows user interface object 10202 unsnapping fromobject placement guide 10208 and tactile output 10213, different fromtactile output 10211 shown in FIG. 5B, is generated on touch-sensitivesurface 451.

In some embodiments, performing the operation includes snapping (10324)the user interface object (e.g., a picture, text box, shape or someother moveable user interface object) into a respective object placementguide and reversing the operation includes snapping the user interfaceobject out of the respective object placement guide. In someembodiments, snapping a user interface object into a respective objectplacement guide includes detecting user-controlled movement of the userinterface object within a predefined distance from the respective objectplacement guide and, in response to detecting the user-controlledmovement of the user interface object within the predefined distance ofthe respective object placement guide, automatically moving (e.g., viadevice-controlled movement) the respective user interface objectadjacent to the respective object placement guide. In some embodiments,once the user interface object has been snapped into a respective objectplacement guide, subsequent movement of the contact across thetouch-sensitive surface does not cause movement of the user interfaceobject until a predefined precondition is met. In particular, in someembodiments, snapping a user interface object out of a respective objectplacement guide includes detecting movement of a contact that wouldcorrespond to user-controlled movement of the user interface object morethan a predefined distance away from the respective object placementguide if the object were not snapped to the respective object placementguide and in response to detecting the movement of the contact,automatically moving (e.g., via device-controlled movement) therespective user interface object to a location on the display that isaway from the respective object placement guide in accordance with themovement of the contact on the touch-sensitive surface. For example,FIGS. 5A-5C show user interface object 10202 snapping to objectplacement guide 10208 (FIGS. 5A-5B), then snapping user interface object10202 out of object placement guide 10208 (FIGS. 5B-5C).

In some embodiments, performing the operation includes marking (10326)data corresponding to the user interface object for deletion (e.g.,placing an icon corresponding to a file in a trash or recycle folder)and reversing the operation includes unmarking the data corresponding tothe user interface object for deletion (e.g., removing/restoring an iconcorresponding to a file from a trash or recycle folder). FIGS. 5D-5F,for example, show user interface object 10222 moving over trash folder10228 and causing the device to, in response, mark data corresponding touser interface object 10222 for deletion (FIGS. 5D-5E) then moving userinterface object 10222 away from trash folder 10228 and causing thedevice to, in response, unmark data corresponding to user interfaceobject 10222 for deletion (FIGS. 5E-5F).

In some embodiments, the user interface object corresponds to a file,performing the operation includes placing (10328) the file in adirectory and reversing the operation includes removing the file fromthe directory. For example, in some embodiments, performing theoperation includes moving a user interface object that is a graphicalrepresentation of a file to a location corresponding to a folder iconthat represents the directory and reversing the operation includesremoving the icon from a graphical representation of thefolder/directory. FIGS. 5G-5I, for example, show user interface object10242, corresponding to a file, moving over folder 10248 and causing thedevice to, in response, place the file in a directory represented byfolder 10248 (FIGS. 5G-5H) then moving user interface object 10242 awayfrom folder 10248 causing the device to, in response, remove the filefrom the directory (FIGS. 5H-5I).

In some embodiments, the user interface object corresponds to anapplication, performing the operation includes placing (10330) the userinterface object in an application launch region and reversing theoperation includes removing the user interface object from theapplication launch region. Examples of an application launch regioninclude a dock or a quick launch bar. FIGS. 5J-5L, for example, showuser interface object 10262, corresponding to an application, movingover application launch region 10268 and causing the device to, inresponse, place user interface object 10262 in application launch region10268 (FIGS. 5J-5K) then moving user interface object 10262 away fromapplication launch region 10268 and causing the device to, in response,remove user interface object 10262 from application launch region 10268(FIGS. 5K-5L).

In response to detecting (10320) the second movement, in addition toreversing the operation, the device generates (10332) a second tactileoutput on the touch-sensitive surface, where the second tactile outputis different from the first tactile output. As a result, the deviceundoes the previously performed operation. In some embodiments the firsttactile output is different from the second tactile output based ondifferences in amplitudes of the tactile outputs. In some embodiments,the first type of tactile output is generated by movement of thetouch-sensitive surface that includes a first dominant movementcomponent. For example, the generated movement corresponds to an initialimpulse of the first tactile output, ignoring any unintended resonance.In some embodiments, the second type of tactile output is generated bymovement of the touch-sensitive surface that includes a second dominantmovement component. For example, the generated movement corresponds toan initial impulse of the second tactile output, ignoring any unintendedresonance. In some embodiments, the first dominant movement componentand the second dominant movement component have (10334) a same movementprofile and different amplitudes. For example, the first dominantmovement component and the second dominant movement component have thesame movement profile when the first dominant movement component and thesecond dominant movement component have a same waveform shape, such assquare, sine, sawtooth or triangle, and approximately the same period.

In some embodiments the first tactile output is different from thesecond tactile output based on differences in movement profiles of thetactile outputs. In some embodiments, the first type of tactile outputis generated by movement of the touch-sensitive surface that includes afirst dominant movement component. For example, the generated movementcorresponds to an initial impulse of the first tactile output, ignoringany unintended resonance. In some embodiments, the second type oftactile output is generated by movement of the touch-sensitive surfacethat includes a second dominant movement component. For example, thegenerated movement corresponds to an initial impulse of the secondtactile output, ignoring any unintended resonance. In some embodiments,the first dominant movement component and the second dominant movementcomponent have (10336) different movement profiles and a same amplitude.For example, the first dominant movement component and the seconddominant movement component have different movement profiles when thefirst dominant movement component and the second dominant movementcomponent have a different waveform shape, such as square, sine,sawtooth or triangle, and/or approximately the same period.

It should be understood that the particular order in which theoperations in FIGS. 6A-6C have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein are alsoapplicable in an analogous manner to method 10300 described above withrespect to FIGS. 6A-6C. For example, the contacts, movements, userinterface objects, focus selectors, and tactile outputs described abovewith reference to method 10300 optionally have one or more of thecharacteristics of contacts, movements, user interface objects, focusselectors, and tactile outputs described herein with reference to othermethods described herein. For brevity, these details are not repeatedhere.

In accordance with some embodiments, FIG. 7 shows a functional blockdiagram of an electronic device 10400 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 7 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 7, an electronic device 10400 includes a display unit10402 configured to display a user interface object; a touch-sensitivesurface unit 10404 configured to detect user contacts; and a processingunit 10406 coupled to display unit 10402 and touch-sensitive surfaceunit 10404. In some embodiments, the processing unit includes adetecting unit 10408, a display enabling unit 10410, a performing unit10412, a generating unit 10414, and a reversing unit 10416.

The processing unit 10406 is configured to detect a contact on thetouch-sensitive surface unit, detect a first movement of the contactacross the touch-sensitive surface unit (e.g., with detecting unit10408), the first movement corresponding to performing an operation onthe user interface object, and in response to detecting the firstmovement; perform the operation (e.g., with performing unit 10412) andgenerate a first tactile output on the touch-sensitive surface unit(e.g., with generating unit 10414). The processing unit 10406 is furtherconfigured to detect a second movement of the contact across thetouch-sensitive surface unit (e.g., with detecting unit 10408), thesecond movement corresponding to reversing the operation on the userinterface object; and in response to detecting the second movement,reverse the operation (e.g., with reversing unit 10416) and generate asecond tactile output on the touch-sensitive surface unit (e.g., withgenerating unit 10414), where the second tactile output is differentfrom the first tactile output.

In some embodiments, the first movement of the contact across thetouch-sensitive surface unit and the second movement of the contactacross the touch-sensitive surface unit are part of a single continuousgesture performed without detecting a liftoff of the contact from thetouch-sensitive surface unit 10404.

In some embodiments, performing the operation (e.g., with the performingunit 10412) includes snapping the user interface object into arespective object placement guide and reversing the operation (e.g.,with the reversing unit 10416) includes snapping the user interfaceobject out of the respective object placement guide.

In some embodiments, performing the operation (e.g., with the performingunit 10412) includes marking data corresponding to the user interfaceobject for deletion and reversing the operation (e.g., with reversingunit 10416) includes unmarking the data corresponding to the userinterface object for deletion.

In some embodiments, the user interface object corresponds to a file,performing the operation (e.g., with the performing unit 10412) includesplacing the file in a directory and reversing the operation (e.g., withthe reversing unit 10416) includes removing the file from the directory.

In some embodiments, the user interface object corresponds to anapplication, performing the operation (e.g., with the performing unit10412) includes placing the user interface object in an applicationlaunch region and reversing the operation (e.g., with the reversing unit10416) includes removing the user interface object from the applicationlaunch region.

In some embodiments, the first tactile output is generated (e.g., withthe generating unit 10414) by movement of the touch-sensitive surfaceunit that includes a first dominant movement component, the secondtactile output is generated (e.g., with the generating unit 10414) bymovement of the touch-sensitive surface unit that includes a seconddominant movement component, and the first dominant movement componentand the second dominant movement component have a same movement profileand different amplitudes.

In some embodiments, the first tactile output is generated (e.g., withthe generating unit 10414) by movement of the touch-sensitive surfaceunit that includes a first dominant movement component, the secondtactile output is generated (e.g., with the generating unit 10414) bymovement of the touch-sensitive surface unit that includes a seconddominant movement component, and the first dominant movement componentand the second dominant movement component have different movementprofiles and a same amplitude.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 6A-6C are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG. 7.For example, detection operations 10304 and 10306, performing operation10312, generating operations 10314 and 10332, and reversing operation10322 are, optionally, implemented by event sorter 170, event recognizer180, and event handler 190. Event monitor 171 in event sorter 170detects a contact on touch-sensitive display 112, and event dispatchermodule 174 delivers the event information to application 136-1. Arespective event recognizer 180 of application 136-1 compares the eventinformation to respective event definitions 186, and determines whethera first contact at a first location on the touch-sensitive surfacecorresponds to a predefined event or sub-event, such as selection of anobject on a user interface. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally utilizes or calls data updater 176 or objectupdater 177 to update the application internal state 192. In someembodiments, event handler 190 accesses a respective GUI updater 178 toupdate what is displayed by the application. Similarly, it would beclear to a person having ordinary skill in the art how other processescan be implemented based on the components depicted in FIGS. 1A-1B.

Indicating Changes in the Z-Order of User Interface Objects

Many electronic devices display user interface objects that have a layerorder (e.g., a z-order or front-to-back order of the user interfaceobjects). A user typically interacts with such objects by repositioningthem on the display, and overlapping objects are displayed on thedisplay in accordance with their front-to-back order (e.g., an objectthat is “in front” of another object is displayed where the two objectsoverlap). In addition to repositioning the objects on the display, auser often wants to change the front-to-back order of the objects on thedisplay. In some methods, changes in the z-order are indicated visually.The embodiments described below improve on these methods by providingfor tactile outputs when objects overlap each other and their z-orderchanges. Thus, the user has tactile as well as visual indication of thechange in z-order when the objects overlap, and thus their covering ofeach other changes with the change in z-order.

FIGS. 8A-8S illustrate exemplary user interfaces for indicating changesin the z-order of user interface objects in accordance with someembodiments. The user interfaces in these figures are used to illustratethe processes described below, including the processes in FIGS. 9A-9D.FIGS. 8A-8S include intensity diagrams that show the current intensityof the contact on the touch-sensitive surface relative to a plurality ofintensity thresholds including a light press intensity threshold (e.g.,“IT_(L)”) and a deep press intensity threshold (e.g., “IT_(D)”). In someembodiments, operations similar to those described below with referenceto “IT_(D)” are performed with reference to a different intensitythreshold (e.g., “IT_(L)”).

FIG. 8A shows user interface objects 10506-1 and 10506-2 displayed ondisplay 450 (e.g., display 340, touch screen 112) of a device (e.g.,device 300, 100). Objects 10506 are, optionally, windows of respectiveapplications, shapes or other graphics in a drawing, or objects (e.g.,text block, picture, etc.) in a presentation. Objects 10506-1 and10506-2 are displayed with a z-order. In FIG. 8A, object 10506-1 is infront of (or “above”) object 10506-2 in the z-order, and, likewise,object 10506-2 is in back of (or “behind” or “below”) object 10506-1 inthe z-order. If two objects do not overlap (e.g., objects 10506-1 and10506-2 in FIG. 8A), their z-order relative to each other may not bevisually displayed to a user. In FIG. 8D, object 10506-1 is in front ofobject 10506-2 in the z-order and that relative z-order is visuallydisplayed to the user, as objects 10506-1 and 10506-2 overlap, andobject 10506-1 covers at least a part of object 10506-2.

FIG. 8A also shows cursor 10504 displayed on display 450. Cursor 10504is an example of a focus selector. A user optionally positions cursor10504 over an object 10506 to bring that object into focus. In FIG. 8A,cursor 10504 is located over object 10506-1.

FIG. 8B shows contact 10510 detected on touch-sensitive surface 451.While contact 10510 is detected on touch-sensitive surface 451, arequest to move object 10506-1 below object 10506-2 in the z-order isreceived by the device (e.g., as shown in FIG. 8C). The deviceoptionally receives the request in the form of, for example, a gestureinput performed on touch-sensitive surface 451 (e.g., a gestureperformed with contact 10510) while cursor 10504 is located over object10506-1, an increase in the intensity of contact 10510 above anintensity threshold while cursor 10504 is located over object 10506-1,or an input made using a keyboard or other input device (e.g., akeyboard shortcut, a selection of a menu option using the keyboard orother input device). The intensity (and the change in intensity) is,optionally, detected by one or more sensors, included in the device,that are configured to detect intensity of contacts with touch-sensitivesurface 451. In some embodiments, while contact 10510 has an intensitybetween IT_(L) and IT_(D), the user is enabled to move the objectassociated with cursor 10504 by moving contact 10510 on thetouch-sensitive surface.

In response to the request (e.g., the increase in intensity of contactfrom an intensity below IT_(D) in FIG. 8B to an intensity above IT_(D)in FIG. 8C), object 10506-1 is moved below object 10506-2 in thez-order. The change in z-order is, optionally, not visually displayed tothe user if objects 10506-1 and 10506-2 do not overlap, as shown in FIG.8C. In accordance with a determination that objects 10506-1 and 10506-2do not overlap, no tactile output associated with the move of object10506-1 below object 10506-2 is generated.

FIG. 8D shows objects 10506-1 and 10506-2 displayed, in a z-order inwhich object 10506-1 is in front of object 10506-2, on display 450. InFIG. 8D, objects 10506-1 and 10506-2 overlap, with object 10506-1covering a part of object 10506-2. Cursor 10504 is displayed as locatedover object 10506-1. For example, in FIG. 8D, the device detectedmovement 10511 of contact 10512 down and to the right on thetouch-sensitive surface 451 while cursor 10504 was over object 10506-1and the intensity of contact 10512 was between IT_(L) and IT_(D), and inresponse to detecting the movement 10511 of contact 10512, the devicemoved cursor 10504 and object 10506-1 down and to the left on thedisplay 450 in accordance with the movement of contact 10512 on thetouch-sensitive surface.

FIG. 8D also shows contact 10512 detected on touch-sensitive surface451. While contact 10512 is detected on touch-sensitive surface 451, arequest to move object 10506-1 below object 10506-2 in the z-order isreceived by the device (e.g., as shown in FIG. 8C). The deviceoptionally receives the request in the form of, for example, a gestureinput performed on touch-sensitive surface 451 (e.g., a gestureperformed with contact 10512) while cursor 10504 is located over object10506-1, an increase in the intensity of contact 10512 above theintensity threshold while cursor 10504 is located over object 10506-1,or an input made using a keyboard or other input device (e.g., akeyboard shortcut, a selection of a menu option using the keyboard orother input device).

In response to the request (e.g., the increase in intensity of contactfrom an intensity below IT_(D) in FIG. 8D to an intensity above IT_(D)in FIG. 8E), object 10506-1 is moved below object 10506-2 in thez-order, as shown in FIG. 8E. With the change in z-order, object 10506-2now covers a part of object 10506-1. In accordance with a determinationthat objects 10506-1 and 10506-2 overlap, a tactile output 10513associated with the movement of object 10506-1 below object 10506-2 isgenerated in conjunction with the move of object 10506-1 below object10506-2. The tactile output may be sensed by the user via contact 10512as a tactile sensation. In some embodiments, the tactile output isgenerated by movement of touch-sensitive surface 451, and the movementincludes a dominant movement component, which optionally has a waveformshape with a wavelength, such as a square, sine, squine, sawtooth, ortriangle.

Many electronic devices display user interface objects that have a layerorder (e.g., a z-order or front-to-back order of the user interfaceobjects). A user typically interacts with such objects by repositioningthem on the display, and overlapping objects are displayed on thedisplay in accordance with their front-to-back order (e.g., an objectthat is “in front” of another object is displayed where the two objectsoverlap). In addition to repositioning the objects on the display, auser often wants to change the front-to-back order of the objects on thedisplay.

Thus, when the relative z-order between two objects is changed, atactile output is generated if the two objects overlap (e.g., 10513 inFIG. 8E), and a tactile output is not generated if the two objects donot overlap (e.g., as shown in FIG. 8C). The tactile output gives theuser an indication that the change in z-order affects which object iscovered by the other object.

FIGS. 8F-8I illustrate an example of moving an object within the z-orderusing a control for changing the z-order. FIG. 8F shows objects 10506-1and 10506-2, and cursor 10504, displayed on display 450. Objects 10506-1and 10506-2 are displayed in a z-order, with object 10506-1 in front ofobject 10506-2 in the z-order. Z-order slider 10514 is also displayed ondisplay 450. Z-order slider 10514 includes slider thumbs 10516-1 and10516-2. Thumb 10516-1 corresponds to object 10506-1 and thumb 10516-2corresponds to object 10506-2. The position of a thumb 10516 relative tothe other thumbs on z-order slider 10514 corresponds to thecorresponding object's position in the z-order. For example, as depictedin FIG. 8F, the further left a thumb 10516 is on z-order slider 10514,the further up front the corresponding object 10506 is in the z-order.Thus, thumb 10516-1 is to the left of thumb 10516-2, corresponding tothe z-order of objects 10506-1 and 10506-2 as shown. It should beappreciated that the correspondence between being further left onz-order slider 10514 and the corresponding object being further up frontin the z-order is a design choice; optionally, the further right a thumb10516 is on z-order slider 10514, the further up front the correspondingobject 10506 is in the z-order.

In FIG. 8F, the device detects movement 10517 of contact 10518 downwardon the touch-sensitive surface 451, and in response to detecting themovement 10517 of contact 10518, the device moves cursor 10504 overthumb 10516-1. FIG. 8G also shows contact 10518 detected ontouch-sensitive surface 451 while cursor 10504 is located over thumb10516-1. The gesture including contact 10518 includes movement 10519 ofa contact 10518 on touch-sensitive surface 451 while contact 10518 hasan intensity between IT_(L) and IT_(D). In response to detection of thegesture including movement 10519 of contact 10518, thumb 10516-1 ismoved rightward on slider 10514 past thumb 10516-2, so that thumb10516-2 is to the left of thumb 10516-1 on slider 10514, as shown inFIG. 8G. In response to the movement of thumb 10516-1 to the right ofthumb 10516-2, object 10506-1, which corresponds to thumb 10516-1, ismoved downward, below object 10506-2 in the z-order. In accordance witha determination that objects 10506-1 and 10506-2 overlap, a tactileoutput 10513 associated with the move of object 10506-1 below object10506-2 is generated in conjunction with the move of object 10506-1below object 10506-2. The tactile output may be sensed by the user viacontact 10518 as a tactile sensation. In some embodiments, the tactileoutput is generated by movement of touch-sensitive surface 451, and themovement includes a dominant movement component, which optionally has awaveform shape, such as a square, sine, squine, sawtooth, or triangle.

FIG. 8I shows objects 10506-1 and 10506-2 and slider 10514 displayed ondisplay 450, with object 10506-1 in front of object 10506-2 in thez-order. In FIG. 8H, the device detects movement 10521 of contact 10520downward and to the right on the touch-sensitive surface 451, and inresponse to detecting the movement 10521 of contact 10520, the devicemoves cursor 10504 over thumb 10516-2. FIG. 8H also shows cursor 10504,located over thumb 10516-2, displayed on display 450, and a gestureincluding movement 10523 of contact 10520 to the left detected ontouch-sensitive surface 451 while cursor 10504 is located over thumb10516-2. The gesture including contact 10520 includes movement 10523 ofa contact on touch-sensitive surface 451. In response to detection ofthe gesture including contact 10520, thumb 10516-2 is moved leftward onslider 10514 past thumb 10516-1, so that thumb 10516-2 is to the left ofthumb 10516-1 on slider 10514, as shown in FIG. 8I. In response to themovement of thumb 10516-2 to the left of thumb 10516-1, object 10506-2,which corresponds to thumb 10516-2, is moved upward, in front of object10506-1 in the z-order. In accordance with a determination that objects10506-1 and 10506-2 overlap, a tactile output 10513 associated with themove of object 10506-2 above object 10506-1 is generated in conjunctionwith the move of object 10506-2 above object 10506-1. The tactile outputmay be sensed by the user via contact 10520 as a tactile sensation. Insome embodiments, the tactile output is generated by movement oftouch-sensitive surface 451, and the movement includes a dominantmovement component, which optionally has a waveform shape, such as asquare, sine, squine, sawtooth, or triangle.

In some embodiments, the tactile output associated with the move ofobject 10506-1 below object 10506-2 in the z-order has a wavelength thatis determined based on a position of object 10506-2 in the z-order priorto receiving the request to move object 10506-1 below object 10506-2 inthe z-order.

FIG. 8J shows user interface objects 10506-1, 10506-2, and 10506-3displayed on display 450. Objects 10506-1, 10506-2, and 10506-3 aredisplayed with a z-order. In FIG. 8J, object 10506-1 is in front ofobjects 10506-2 and 10506-3. Object 10506-3 is in front of object10506-2 but in back of object 10506-1. Thus, object 10506-3 is anintervening object between objects 10506-1 and 10506-2 in the z-order,even though there is no visual indication of this ordering in FIG. 8J.Objects 10506-1 and 10506-3 do not overlap, and objects 10506-1 and10506-2 overlap. Cursor 10504 is displayed over object 10506-1.

FIG. 8J shows contact 10524 detected on touch-sensitive surface 451.While contact 10524 is detected on touch-sensitive surface 451, arequest to move object 10506-1 below object 10506-2 in the z-order isreceived by the device (e.g., an increase in intensity of contact 10524from an intensity below IT_(L) in FIG. 8J to an intensity above IT_(D)in FIG. 8L). The device optionally receives the request in the form of,for example, a gesture input performed on touch-sensitive surface 451(e.g., a gesture performed with contact 10524) while cursor 10504 islocated over object 10506-1, an increase in the intensity of contact10524 above the intensity threshold while cursor 10504 is located overobject 10506-1, or an input made using a keyboard or other input device(e.g., a keyboard shortcut, a selection of a menu option using thekeyboard or other input device).

In response to the request, object 10506-1 is moved below interveningobject 10506-3 (e.g., when contact 10524 reaches an intensity aboveIT_(L) in FIG. 8K) and then below object 10506-2 (e.g., when contact10524 reaches an intensity above IT_(D) in FIG. 8M) in the z-order, asshown in FIG. 8K-8L. When object 10506-1 is moved below object 10506-3,in accordance with a determination that objects 10506-1 and 10506-3 donot overlap, no tactile output associated with the move of object10506-1 below object 10506-3 is generated. When object 10506-1 is movedbelow object 10506-2, in accordance with a determination that objects10506-1 and 10506-2 overlap, a tactile output 10525 associated with themove of object 10506-1 below object 10506-2 is generated in conjunctionwith the move of object 10506-1 below object 10506-2. The tactile outputmay be sensed by user via contact 10524 as a tactile sensation. In someembodiments, the tactile output is generated by movement oftouch-sensitive surface 451, and the movement includes a dominantmovement component, which optionally has a waveform shape, such as asquare, sine, squine, sawtooth, or triangle.

FIG. 8M shows user interface objects 10506-1, 10506-2, and 10506-3displayed on display 450. Objects 10506-1, 10506-2, and 10506-3 aredisplayed with a z-order. In FIG. 8M, object 10506-1 is in front ofobjects 10506-2 and 10506-3. Object 10506-3 is in front of object10506-2 but in back of object 10506-1. Thus, object 10506-3 is anintervening object between objects 10506-1 and 10506-2 within thez-order. Objects 10506-1, 10506-2, and 10506-3 overlap. Cursor 10504 isdisplayed over object 10506-1.

FIG. 8M shows contact 10526 detected on touch-sensitive surface 451.While contact 10526 is detected on touch-sensitive surface 451, arequest to move object 10506-1 below object 10506-2 in the z-order isreceived by the device (e.g., an increase in intensity of contact 10526from an intensity below IT_(L) in FIG. 8M to an intensity above IT_(D)in FIG. 8O). In some circumstances, the device receives the request inthe form of, for example, a gesture input performed on touch-sensitivesurface 451 (e.g., a gesture performed with contact 10526) while cursor10504 is located over object 10506-1, an increase in the intensity ofcontact 10526 above the intensity threshold while cursor 10504 islocated over object 10506-1, or an input made using a keyboard or otherinput device (e.g., a keyboard shortcut, a selection of a menu optionusing the keyboard or other input device).

In response to the request, object 10506-1 is moved below object 10506-3(e.g., when contact 10524 reaches an intensity above IT_(L) in FIG. 8N)and then below object 10506-2 (e.g., when contact 10524 reaches anintensity above IT_(D) in FIG. 8O) in the z-order, as shown in FIGS.8N-8O. When object 10506-1 is moved below object 10506-3, in accordancewith a determination that objects 10506-1 and 10506-3 overlap, a tactileoutput 10527 associated with the move of object 10506-1 below object10506-3 is generated in conjunction with the move of object 10506-1below object 10506-3. This tactile output associated with the move ofobject 10506-1 below object 10506-3 may be sensed by user via contact10526 as a tactile sensation. When object 10506-1 is moved below object10506-2, in accordance with a determination that objects 10506-1 and10506-2 overlap, a tactile output 10528 associated with the move ofobject 10506-1 below object 10506-2 is generated in conjunction with themove of object 10506-1 below object 10506-2. This tactile outputassociated with the move of object 10506-1 below object 10506-2 may besensed by user via contact 10526 as a tactile sensation. In someembodiments, the tactile outputs are generated by respective movementsof touch-sensitive surface 451, and the respective movements eachinclude a dominant movement component, which optionally has a waveformshape, such as a square, sine, squine, sawtooth, or triangle.

In some embodiments, the tactile output 10527 associated with the moveof object 10506-1 below object 10506-3 (hereinafter “Tactile Output A”)and the tactile output 10528 associated with the move of object 10506-1below object 10506-2 after moving below object 10506-3 (hereinafter“Tactile Output B”) are different. For example, the dominant movementcomponent for Tactile Output A 10527 optionally has a differentwavelength than the dominant movement component for Tactile Output B10528. In some embodiments, the wavelength for Tactile Output A 10527 isdetermined based on a position of object 10506-3 in the z-order, and thewavelength for Tactile Output B 10528 is determined based on a positionof object 10506-2 in the z-order.

In some embodiments, the wavelength of Tactile Output A 10527 isdetermined based on a number of user interface objects 10506, thatobject 10506-1 overlaps, that are between object 10506-1 and object10506-3 in the z-order. In some embodiments, the wavelength of TactileOutput B 10528 is determined based on a number of user interface objects10506, that object 10506-1 overlaps, that are between object 10506-1 andobject 10506-2 in the z-order.

Object 10506-1 optionally overlaps multiple other user interface objectsarranged in a respective z-order sequence, irrespective of whetherobject 10506-1 overlaps with object 10506-2. Thus, in some embodiments,the z-order includes object 10506-1, the multiple other user interfaceobjects behind object 10506-1, and then object 10506-2. Thus, whenobject 10506-1 is moved below object 10506-2 in the z-order, inaccordance with a request to move object 10506-1 below object 10506-2 inthe z-order, object 10506-1 is moved below each of the multiple otheruser interface objects in sequence before being moved below object10506-2. For each of the multiple other user interface objects thatobjects 10506-1 moves below, a tactile output is generated inconjunction with the move of object 10506-1 below the respective userinterface object. Thus, as object 10506-1 is moved below the multipleother user interface objects, a sequence of tactile outputs isgenerated. In some embodiments, the sequence of tactile outputs isgenerated based on a mathematical progression. For example, eachsuccessive tactile output has a wavelength that is double the wavelengthof the preceding tactile output. In some other embodiments, the sequenceof tactile outputs is generated based on a musical progression. Forexample, each successive tactile output corresponds to a next note in apredefined musical scale or the same note in a lower octave.

FIGS. 8P-8S illustrate an example of the user interfaces described abovewith reference to FIGS. 8A-8O implemented on a device with atouch-sensitive display (e.g., device 100 with touch screen 112). FIGS.8P-8Q show objects 10532-1 and 10532-2 displayed on touch-sensitivedisplay 112. Objects 10532-1 and 10532-2, which do not overlap, aredisplayed in a z-order, with object 10532-1 in front of object 10532-2in the z-order.

FIG. 8P shows contact 10534 detected on touch-sensitive display 112 at aposition over object 10532-1. While contact 10534 is detected ontouch-sensitive display 112, a request to move object 10532-1 belowobject 10532-2 in the z-order is received by the device. The deviceoptionally receives the request in the form of, for example, a gestureinput performed on touch-sensitive display 112 (e.g., a gestureperformed with contact 10534 over object 10532-1 or with another contacton touch-sensitive display 112) or an increase in the intensity ofcontact 10534 over object 10532-1 above the intensity threshold (e.g.,an increase in intensity of contact 10534 from an intensity below IT_(D)in FIG. 8P to an intensity above IT_(D) in FIG. 8Q).

In response to the request, object 10532-1 is moved below object 10532-2in the z-order, as shown in FIG. 8Q. In accordance with a determinationthat objects 10532-1 and 10532-2 do not overlap, no tactile outputassociated with the move of object 10532-1 below object 10532-2 isgenerated.

FIG. 8R shows overlapping objects 10532-1 and 10532-2 displayed ontouch-sensitive display 112. Objects 10532-1 and 10532-2 are displayedin a z-order, with object 10532-1 in front of object 10532-2 in thez-order. FIG. 8R also shows contact 10536 detected on touch-sensitivedisplay 112 at a position over object 10532-1. While contact 10536 isdetected on touch-sensitive display 112, a request to move object10532-1 below object 10532-2 in the z-order is received by the device.The device optionally receives the request in the form of, for example,a gesture input performed on touch-sensitive display 112 (e.g., agesture performed with contact 10536 over object 10532-1 or with anothercontact on touch-sensitive display 112) or an increase in the intensityof contact 10536 over object 10532-1 above the intensity threshold(e.g., an increase in intensity of contact 10536 from an intensity belowIT_(D) in FIG. 8R to an intensity above IT_(D) in FIG. 8S).

In response to the request, object 10532-1 is moved below object 10532-2in the z-order, as shown in FIG. 8S. In accordance with a determinationthat objects 10532-1 and 10532-2 overlap, a tactile output 10537associated with the move of object 10532-1 below object 10532-2 isgenerated in conjunction with the move of object 10532-1 below object10532-2. The tactile output may be sensed by user via contact 10536 as atactile sensation. In some embodiments, the tactile output is generatedby movement of touch-sensitive display 112, and the movement includes adominant movement component, which optionally has a waveform shape, suchas a square, sine, squine, sawtooth, or triangle. In some embodiments,the tactile output associated with the move of object 10532-1 belowobject 10532-2 in the z-order has a wavelength that is determined basedon a position of object 10532-2 in the z-order prior to receiving therequest to move object 10532-1 below object 10532-2 in the z-order.

FIGS. 9A-9D are flow diagrams illustrating a method 10600 of indicatingchanges in the z-order of user interface objects in accordance with someembodiments. The method 10600 is performed at an electronic device(e.g., device 300, FIG. 3, or portable multifunction device 100, FIG.1A) with a display and a touch-sensitive surface. In some embodiments,the display is a touch screen display and the touch-sensitive surface ison the display. In some embodiments, the display is separate from thetouch-sensitive surface. Some operations in method 10600 are,optionally, combined and/or the order of some operations is, optionally,changed.

As described below, the method 10600 provides an intuitive way toindicate changes in the z-order of user interface objects. The methodreduces the cognitive burden on a user when indicating changes in thez-order of user interface objects, thereby creating a more efficienthuman-machine interface. For battery-operated electronic devices,enabling a user to perceive changes in the z-order of user interfaceobjects faster and more efficiently conserves power and increases thetime between battery charges.

The device displays (10602) a plurality of user interface objects on thedisplay, where the plurality of user interface objects have a z-order,the plurality of user interface objects includes a first user interfaceobject and a second user interface object, and the first user interfaceobject is above the second user interface object in the z-order.Multiple user interface objects, such as objects 10506-1 and 10506-2, orobjects 10532-1 and 10532-2, are, optionally, displayed with a z-order,as shown in FIG. 8A, 8D, or 8P, respectively. In FIGS. 8A and 8D, object10506-1 is above object 10506-2 in the z-order. In FIG. 8P, object10532-1 is above object 10532-2 in the z-order.

While detecting a contact (e.g., a finger contact) on thetouch-sensitive surface, the device receives (10604) a request to movethe first user interface object below the second user interface objectin the z-order. For example, while a contact (e.g., contact 10510, FIG.8A; contact 10512, FIG. 8D; contact 10518, FIG. 8F; contact 10520, FIG.8H; contact 10524, FIG. 8J; contact 10526, FIG. 8M; contact 10534, FIG.8Q; contact 10536, FIG. 8R), is detected on the touch-sensitive surface(e.g., touch-sensitive surface 451, touch-sensitive display 112), arequest to move object 10506-1 below object 10506-2 is received.

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes(10606), while a focus selector is over the first user interface object,detecting an increase in intensity of the contact above a respectiveintensity threshold (e.g., the deep press intensity threshold IT_(D))Thus, the user is intuitively enabled to press the first user interfaceobject “down” below the second user interface object in the z-order. Forexample, while contact cursor 10504 is located over object 10506-1, theintensity of contact 10510 (or contact 10512 or 10524 or 10526) is,optionally, increased from an intensity below IT_(D) to an intensityabove IT_(D). In embodiments where the touch-sensitive surface is atouch-sensitive display (e.g., touch-sensitive display 112), receivingthe request includes detecting an increase in intensity of the contact(e.g., an increase in intensity of contact 10534 or 10536 above the deeppress intensity threshold IT_(D)) while the contact is over the firstuser object (e.g., object 10532-1).

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes(10608), while displaying a control for changing a z-order of the firstuser interface object (e.g., a slider or other control that isseparate/distinct from the first user interface object that determines az-order of the first user interface object), detecting an input on thecontrol that corresponds to moving the first user interface objectdownward in the z-order. The control for changing z-order is,optionally, slider 10514 (FIG. 8F). While z-order slider 10514 isdisplayed, an input corresponding to moving object 10506-1 downward(e.g., the gesture including movement 10519 of contact 10518 ontouch-sensitive surface 451 while cursor 10504 is over slider thumb10516-1) is detected. In response to detecting the gesture includingmovement 10519 of contact 10518 the device moves thumb 10516-1, whichcorresponds to object 10506-1, on slider 10514 that corresponds tomoving object 10506-1 downward in the z-order.

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes(10610), while displaying a control for changing a z-order of the seconduser interface object (e.g., a slider or other control that isseparate/distinct from the second user interface object that determinesa z-order of the first user interface object), detecting an input on thecontrol that corresponds to moving the second user interface objectupward in the z-order. The control for changing z-order is, optionally,slider 10514 (FIG. 8H). While z-order slider 10514 is displayed, aninput corresponding to moving object 10506-2 upward (e.g., the gestureincluding movement 10523 of contact 10520 on touch-sensitive surface 451while cursor 10504 is over slider thumb 10516-2) is detected. Inresponse to detecting the gesture including movement 10523 of contact10520, the device moves thumb 10516-2, which corresponds to object10506-2, on slider 10514 that corresponds to moving object 10506-2upward in the z-order.

In response (10612) to the request, the device moves (10614) the firstuser interface object below the second user interface object in thez-order. In response to the request, object 10506-1 (or 10532-1) ismoved below object 10506-2 (or 10532-2) in the z-order, as shown in FIG.8C (or 8E or 8G or 8I or 8L or 8O or 8Q or 8S).

In accordance with a determination that the first user interface objectoverlaps at least a portion of the second user interface object, thedevice generates (10616) a tactile output (e.g., 10513 in FIG. 8E, 8G or8I; 10525 in FIG. 8L; 10528 in FIG. 8O; or 10537 in FIG. 8S) associatedwith moving the first user interface object below the second userinterface object on the touch-sensitive surface in conjunction withmoving the first user interface object below the second user interfaceobject. If objects 10506-1 and 10506-2 (or objects 10532-1 and 10532-2)overlap, as shown in FIG. 8D-8O or 8R-8S, a tactile output associatedwith the move of object 10506-1 (or object 10532-1) below object 10506-2(or object 10532-2) is generated when object 10506-1 (or object 10532-1)is moved below object 10506-2 (or object 10532-2). In some embodiments,the tactile output associated with moving the first user interfaceobject below the second user interface object has (10618) a wavelengththat is determined based on a position of the second user interfaceobject in the z-order prior to receiving the request to move the firstuser interface object below the second user interface object in thez-order (e.g., the lower the first user interface object is moved inz-order, the lower the pitch of the tactile output). For example, thetactile output associated with moving object 10506-1 below 10506-2 has awavelength that is determined based on a position of object 10506-2 inthe z-order prior to receiving the request to move object 10506-1 belowobject 10506-2 in the z-order.

In accordance with a determination that the first user interface objectdoes not overlap the second user interface object, forgoes (10620)generating the tactile output associated with moving the first userinterface object below the second user interface object. If objects10506-1 and 10506-2 (or objects 10532-1 and 10532-2) do not overlap, asshown in FIG. 8A-8C or 8P-8Q), no tactile output associated with themove of object 10506-1 (or object 10532-1) below object 10506-2 (orobject 10532-2) is generated when object 10506-1 (or object 10532-1) ismoved below object 10506-2 (or object 10532-2).

In some embodiments, the first user interface object overlaps (10622) atleast a portion of the second user interface object when at least aportion of the first user interface object covers at least a portion ofthe second user interface object. In some embodiments, the first userinterface object partially overlaps the second user interface object. Insome embodiments, the first user interface object completely overlapsthe second user interface object (e.g., the first user interface objectcovers all of the second user interface object). For example, in FIG.8A, objects 10506-1 and 10506-2 do not overlap, and in FIG. 8D objects10506-1 and 10506-2 overlap.

In some embodiments, the plurality of user interface objects includes(10624) an intervening user interface object, and the intervening userinterface object has a position in the z-order between the first userinterface object and the second user interface object. As shown in FIGS.8J and 8M, for example, there is, optionally, an intervening object10506-3 between objects 10506-1 and 10506-2 in the z-order. In someembodiments, when there is an intervening user interface object, inresponse (10612) to the request to move the first user interface objectbelow the second user interface object in the z-order, the device moves(10630) the first user interface object below the intervening userinterface object in the z-order prior to moving the first user interfaceobject below the second user interface object in the z-order. Inaccordance with a determination that the first user interface objectoverlaps at least a portion of the intervening user interface object,the device generates (10632), on the touch-sensitive surface, a tactileoutput associated with moving the first user interface object below theintervening user interface object in conjunction with moving the firstuser interface object below the intervening user interface object. Inaccordance with a determination that the first user interface objectdoes not overlap the intervening user interface object, the deviceforgoes generating the tactile output associated with moving the firstuser interface object below the intervening user interface object. Inresponse to the request to move object 10506-1 below object 10506-2 inthe z-order, object 10506-1 is moved below object 10506-3 on the way tobeing moved below object 10506-2 in the z-order, as shown in FIGS.8J-8O. If objects 10506-1 and object 10506-3 do not overlap, as shown inFIGS. 8J-8K, no tactile output associated with the move of objects10506-1 below object 10506-3 is generated. If objects 10506-1 and object10506-3 overlap, as shown in FIGS. 8M-8N, a tactile output associatedwith the move of objects 10506-1 below object 10506-3 is generated.

In some embodiments, the first user interface object overlaps (10635)the intervening user interface object and the second user interfaceobject, and in response (10612) to the request to move the first userinterface object below the second user interface object in the z-order,the device generates (10636) a first tactile output, on thetouch-sensitive surface, associated with moving the first user interfaceobject below the intervening user interface object, prior to moving thefirst user interface object below the second user interface object inthe z-order; and generates (10638) a second tactile output, on thetouch-sensitive surface, associated with moving the first user interfaceobject below the second user interface object, wherein the first tactileoutput is different from the second tactile output. For example, asshown in FIG. 8M, object 10506-1 overlaps with object 10506-3 and object10506-2. In response to the request to move object 10506-1 below object10506-2 in the z-order, object 10506-1 is moved below object 10506-3 andthen moved below object 10506-2 in the z-order. As object 10506-1overlaps with object 10506-3 and object 10506-2, Tactile Output A 10527is generated for the move of object 10506-1 below object 10506-3, andTactile Output B 10528 is generated for the move of object 10506-1 belowobject 10506-2. Tactile Output A 10527 and Tactile Output B 10528 are,optionally, different. For example, Tactile Output A 10527 and TactileOutput B 10528 optionally have different wavelengths or amplitude.

In some embodiments, the first tactile output is (10640) generated bymovement of the touch-sensitive surface that includes a first dominantmovement component (e.g., movement corresponding to an initial impulseof the first tactile output, ignoring any unintended resonance), thesecond tactile output is generated by movement of the touch-sensitivesurface that includes a second dominant movement component (e.g.,movement corresponding to an initial impulse of the second tactileoutput, ignoring any unintended resonance), and the first dominantmovement component and the second dominant movement component havedifferent wavelengths (e.g., while maintaining a same movement profilesuch as a same waveform shape such as square, sine, squine, sawtooth ortriangle and/or while maintaining a same amplitude). Tactile Output A10527 and Tactile Output B 10528 have respective dominant movementcomponents that have different wavelengths. Thus, Tactile Output A 10527for the move below object 10506-3 and Tactile Output B 10528 for themove below object 10506-2 may feel different to the user.

In some embodiments, the wavelength of the first tactile output is(10642) determined based on a position of the intervening user interfaceobject in the z-order, and the wavelength of the second tactile outputis determined based on a position of the second user interface object inthe z-order (e.g., the tactile output that is generated when the firstuser interface object is moved past a respective user interface objectis determined based on an absolute position of the respective userinterface object in the z-order, which provides feedback to the user asto how far the first user interface object has been pushed down into thez-order). In FIGS. 8M-8O, for example, the wavelength of Tactile OutputA 10527 is, optionally, determined based on the absolute position ofobject 10506-3 in the z-order, and the wavelength of Tactile Output B10528 is, optionally, determined based on the absolute position ofobject 10506-2 in the z-order.

In some embodiments, the wavelength of the first tactile output is(10644) determined based on a number of user interface objects that thefirst user interface object overlaps that are between the first userinterface object and the intervening user interface object in thez-order. In FIGS. 8M-8O, for example, the wavelength of Tactile Output A10527 is, optionally, determined based on the number of user interfaceobjects that overlap object 10506-1 and are between object 10506-1 andobject 10506-3 in the z-order.

In some embodiments, the wavelength of the second tactile output is(10646) determined based on a number of user interface objects that thefirst user interface object overlaps that are between the first userinterface object and the second user interface object in the z-order(e.g., the tactile output that is generated when the first userinterface object is moved past a respective user interface object isdetermined based on how many other objects are between the first userinterface object and the respective object, which provides feedback tothe user as to how far the user interface object has been pushed downinto a “local” z-order for objects that are in the same general area ofthe user interface and overlap each other). In FIGS. 8M-8O, for example,the wavelength of Tactile Output B 10528 is, optionally, determinedbased on the number of user interface objects that overlap object10506-1 and are between object 10506-1 and object 10506-2 in thez-order.

In some embodiments, the first user interface overlaps (10648) aplurality of other user interface objects arranged in a respectivez-order sequence and a next tactile output (e.g., apitch/wavelength/intensity of the next tactile output) corresponding tomovement of the first user interface object below a next user interfaceobject in the z-order sequence is based on a mathematical progressionfrom a prior tactile output corresponding to movement of the first userinterface object below a prior user interface object in the z-ordersequence (e.g., each successive tactile output doubles the wavelength ofthe prior tactile output).

In some embodiments, the first user interface overlaps (10650) aplurality of other user interface objects arranged in a respectivez-order sequence and a next tactile output (e.g., apitch/wavelength/intensity of the next tactile output) corresponding tomovement of the first user interface object below a next user interfaceobject in the z-order sequence is based on a musical progression from aprior tactile output corresponding to movement of the first userinterface object below a prior user interface object in the z-ordersequence (e.g., each successive tactile output corresponds to a nextnote in a predefined musical scale or the same note in a lower octave).

For example, in some circumstances, object 10506-1 overlaps multipleother user interface objects arranged in a respective z-order sequenceirrespective of whether object 10506-1 overlaps with object 10506-2.Thus, in some embodiments, the z-order includes object 10506-1, themultiple other user interface objects behind object 10506-1, and thenobject 10506-2. Thus, when object 10506-1 is moved below object 10506-2in the z-order, in accordance with a request to move object 10506-1below object 10506-2 in the z-order, object 10506-1 is moved below eachof the multiple other user interface objects in z-order sequence beforebeing moved below object 10506-2. In this example, for each of themultiple other user interface objects that objects 10506-1 moves below,a tactile output is generated in conjunction with the move of object10506-1 below the respective other user interface object. Thus, asobject 10506-1 is moved below the multiple other user interface objects,a sequence of tactile outputs is generated. In some embodiments, thesequence of tactile outputs is generated based on a mathematicalprogression. For example, each successive tactile output has awavelength that is double the wavelength of the preceding tactileoutput. In some other embodiments, the sequence of tactile outputs isgenerated based on a musical progression. For example, each successivetactile output corresponds to a next note in a predefined musical scaleor the same note in a lower octave.

It should be understood that the particular order in which theoperations in FIGS. 9A-9D have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein are alsoapplicable in an analogous manner to method 10600 described above withrespect to FIGS. 9A-9D. For example, the contacts, gestures, userinterface objects, tactile outputs, intensity thresholds, and focusselectors described above with reference to method 10600 optionally hasone or more of the characteristics of the contacts, gestures, userinterface objects, tactile outputs, intensity thresholds, and focusselectors described herein with reference to other methods describedherein. For brevity, these details are not repeated here.

In accordance with some embodiments, FIG. 10 shows a functional blockdiagram of an electronic device 10700 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 10 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 10, an electronic device 10700 includes a display unit10702 configured to display a plurality of user interface objects on thedisplay unit 10702, where: the plurality of user interface objects havea z-order, the plurality of user interface objects includes a first userinterface object and a second user interface object, and the first userinterface object is above the second user interface object in thez-order; a touch-sensitive surface unit 10704 configured to receivecontacts; and a processing unit 10706 coupled to the display unit 10702and the touch-sensitive surface unit 10704. In some embodiments, theprocessing unit 10706 includes a receiving unit 10708, a moving unit10710, and a generating unit 10712.

The processing unit 10706 is configured to: while detecting a contact onthe touch-sensitive surface unit 10704, receive a request to move thefirst user interface object below the second user interface object inthe z-order (e.g., with the receiving unit 10708); and in response tothe request: move the first user interface object below the second userinterface object in the z-order (e.g., with the moving unit 10710); inaccordance with a determination that the first user interface objectoverlaps at least a portion of the second user interface object,generate a tactile output associated with moving the first userinterface object below the second user interface object on thetouch-sensitive surface unit 10704 in conjunction with moving the firstuser interface object below the second user interface object (e.g., withthe generating unit 10712); and in accordance with a determination thatthe first user interface object does not overlap the second userinterface object, forgo generating the tactile output associated withmoving the first user interface object below the second user interfaceobject (e.g., with the generating unit 10712).

In some embodiments, the first user interface object overlaps at least aportion of the second user interface object when at least a portion ofthe first user interface object covers at least a portion of the seconduser interface object.

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes, whilea focus selector is over the first user interface object, detecting anincrease in intensity of the contact above a respective intensitythreshold.

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes, whiledisplaying a control for changing a z-order of the first user interfaceobject, detecting an input on the control that corresponds to moving thefirst user interface object downward in the z-order.

In some embodiments, receiving the request to move the first userinterface object below the second user interface object includes, whiledisplaying a control for changing a z-order of the second user interfaceobject, detecting an input on the control that corresponds to moving thesecond user interface object upward in the z-order.

In some embodiments, the plurality of user interface objects includes anintervening user interface object, and the intervening user interfaceobject has a position in the z-order between the first user interfaceobject and the second user interface object. The processing unit 10706is configured to: in response to the request to move the first userinterface object below the second user interface object in the z-order:move the first user interface object below the intervening userinterface object in the z-order prior to moving the first user interfaceobject below the second user interface object in the z-order (e.g., withthe moving unit 10710); in accordance with a determination that thefirst user interface object overlaps at least a portion of theintervening user interface object, generate, on the touch-sensitivesurface unit 10704, a tactile output associated with moving the firstuser interface object below the intervening user interface object inconjunction with moving the first user interface object below theintervening user interface object (e.g., with the generating unit10712); and in accordance with a determination that the first userinterface object does not overlap the intervening user interface object,forgo generating the tactile output associated with moving the firstuser interface object below the intervening user interface object (e.g.,with the generating unit 10712).

In some embodiments, the first user interface object overlaps theintervening user interface object and the second user interface object.The processing unit 10706 is configured to: in response to the requestto move the first user interface object below the second user interfaceobject in the z-order: generate a first tactile output, on thetouch-sensitive surface unit 10704, associated with moving the firstuser interface object below the intervening user interface object, priorto moving the first user interface object below the second userinterface object in the z-order (e.g., with the generating unit 10712);and generate a second tactile output, on the touch-sensitive surfaceunit 10704, associated with moving the first user interface object belowthe second user interface object, wherein the first tactile output isdifferent from the second tactile output (e.g., with the generating unit10712).

In some embodiments, the first tactile output is generated by movementof the touch-sensitive surface unit 10704 that includes a first dominantmovement component, the second tactile output is generated by movementof the touch-sensitive surface unit 10704 that includes a seconddominant movement component, and the first dominant movement componentand the second dominant movement component have different wavelengths.

In some embodiments, the wavelength of the first tactile output isdetermined based on a position of the intervening user interface objectin the z-order, and the wavelength of the second tactile output isdetermined based on a position of the second user interface object inthe z-order.

In some embodiments, the wavelength of the first tactile output isdetermined based on a number of user interface objects that the firstuser interface object overlaps that are between the first user interfaceobject and the intervening user interface object in the z-order.

In some embodiments, the wavelength of the second tactile output isdetermined based on a number of user interface objects that the firstuser interface object overlaps that are between the first user interfaceobject and the second user interface object in the z-order.

In some embodiments, the tactile output associated with moving the firstuser interface object below the second user interface object has awavelength that is determined based on a position of the second userinterface object in the z-order prior to receiving the request to movethe first user interface object below the second user interface objectin the z-order.

In some embodiments, the first user interface overlaps a plurality ofother user interface objects arranged in a respective z-order sequenceand a next tactile output corresponding to movement of the first userinterface object below a next user interface object in the z-ordersequence is based on a mathematical progression from a prior tactileoutput corresponding to movement of the first user interface objectbelow a prior user interface object in the z-order sequence.

In some embodiments, the first user interface overlaps a plurality ofother user interface objects arranged in a respective z-order sequenceand a next tactile output corresponding to movement of the first userinterface object below a next user interface object in the z-ordersequence is based on a musical progression from a prior tactile outputcorresponding to movement of the first user interface object below aprior user interface object in the z-order sequence.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 9A-9D are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.10. For example, receiving operation 10604, moving operation 10614,generating operation 10616, and forgoing operation 10620 are,optionally, implemented by event sorter 170, event recognizer 180, andevent handler 190. Event monitor 171 in event sorter 170 detects acontact on touch-sensitive display 112, and event dispatcher module 174delivers the event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface (or whether rotationof the device) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally utilizes or calls data updater 176 or objectupdater 177 to update the application internal state 192. In someembodiments, event handler 190 accesses a respective GUI updater 178 toupdate what is displayed by the application. Similarly, it would beclear to a person having ordinary skill in the art how other processescan be implemented based on the components depicted in FIGS. 1A-1B.

Providing Tactile Feedback Warning a User

Many electronic devices have graphical user interfaces that display userinterface objects that can be manipulated by adjusting one or moreassociated parameter. For example, a graphical user interface optionallydisplays one or more user interface object (e.g., an image, media clip,audio clip, shape, application window, folder, menu or status bar) thatthe user can customize (e.g., enlarge, shrink, crop, rotate, increasevolume, decrease volume or otherwise manipulate a visual or audioparameter) through a user interface (e.g., mouse, touch-sensitivesurface or keyboard). Due to practical considerations, such as sizeconstraints of an associated display, power constraints of an associatedspeaker and inherent properties of the user interface object (e.g.,size, shape, length and volume), predefined adjustment limits arecommonly assigned to these user interface object, limiting the extent towhich their properties can be adjusted. Given the complexity of a userinterface environment where predefined adjustment limits are applied touser interface objects, there is a need to provide feedback that enablesthe user to more efficiently and conveniently adjust the properties ofthese user interface objects with respect to the predefined adjustmentlimits and alert a user when the predefined adjustment limits have beenreached or exceeded.

The embodiments described below provide improved methods and userinterfaces for generating feedback to a user navigating a complex userinterface. More specifically, these methods and user interfaces providetactile feedback to the user when an action will result in theadjustment of a user interface object parameter beyond a predefinedlimit. The tactile feedback warns the user when their action will resultin the adjustment of a user interface object parameter beyond apredefined adjustment limit. In this fashion, the methods and userinterfaces provided below allow the user to more efficiently discernbetween allowed, forbidden and non-recommended parameter adjustments byproviding tactile feedback, instead of or in addition to audible and/orvisual feedback. Some methods for warning a user that a predefinedadjustment limit has been exceeded rely on an audible or visual cue.However, there are many situations (e.g., at work, in a theatre an invarious social situations) where the volume of an electronic device willbe lowered or muted, rendering audible cues ineffective. Advantageously,the methods and user interfaces described below augment or replaceaudible feedback by providing tactile feedback indicating that apredefined adjustment limit has been or will be exceeded, rendering thewarning effective even when the volume of the electronic device has beenlowered or muted.

FIGS. 11A-11T illustrate exemplary user interfaces for providingfeedback when an action will result in the adjustment of a parameterbeyond a predefined limit in accordance with some embodiments. The userinterfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIGS. 12A-12B.

FIG. 11A illustrates exemplary user interface 10808 displaying images10814-10830 and a control icon 10802 (e.g., a thumb or handle of acontrol) for controlling a parameter (e.g., size) of the images inaccordance with some embodiments. In FIG. 11A, user interface 10808 isdisplayed on display 450 of an electronic device that also includestouch-sensitive surface 451 and one or more sensors for detectingintensity of contacts with touch-sensitive surface. In some embodiments,touch-sensitive surface 451 is a touch screen display that is optionallydisplay 450 or a separate display. User interface 10808 displays acontrol icon 10802 for controlling a parameter (e.g., size) associatedwith respective content (e.g., images 10814, 10816, 10818, 10820, 10822,10824, 10826, 10828 and 10830 displayed on user interface 10808). InFIG. 11A, user interface 10808 also displays cursor 10806, controllableby the user through contacts on touch-sensitive surface 451. Forexample, detection of movement of a contact (e.g., a gesture) ontouch-sensitive surface 451 corresponds to movement of cursor 10806 onuser interface 10808. In FIG. 11A, user interface 10808 also displayssizing bar 10804, corresponding to a plurality of sizes for thedisplayed content (e.g., images). In FIG. 11A, the left and rightboundaries of sizing bar 10804 correspond to predefined sizing limitsfor the displayed content (e.g., images). For example, when control icon10802 is moved to the left boundary of sizing bar 10804, the displayedcontent (e.g., images) are displayed at a size corresponding to apredefined minimum size (e.g., the displayed images are shrunk to asmallest allowable size). Likewise, when control icon 10802 is moved tothe right boundary of sizing bar 10804, the displayed content (e.g.,images) are displayed at a size corresponding to a predefined maximumsize (e.g., the displayed images are expanded to a largest allowablesize).

In some embodiments, the device is an electronic device with a separatedisplay (e.g., display 450) and a separate touch-sensitive surface(e.g., touch-sensitive surface 451). In some embodiments, the device isportable multifunction device 100, the display is touch-sensitivedisplay system 112, and the touch-sensitive surface includes tactileoutput generators 167 on the display (FIG. 1A). For convenience ofexplanation, the embodiments described with reference to FIGS. 11A-11Tand FIGS. 12A-12B will be discussed with reference to display 450 and aseparate touch-sensitive surface 451, however analogous operations are,optionally, performed on a device with a touch-sensitive display system112 in response to detecting movement of the contacts described in FIGS.11A-11T on the touch-sensitive display system 112 while displaying theuser interfaces shown in FIGS. 11A-11T on the touch-sensitive displaysystem 112; in such embodiments, the focus selector is, optionally: arespective contact, a representative point corresponding to a contact(e.g., a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112, in place of cursor 10806.

FIGS. 11A-11H illustrate that contact 10810 and a gesture includingmovement of contact 10810 are detected on touch-sensitive surface 451(e.g., movement 10812-a of contact 10810 from location 10810-a in FIG.11A to location 10810-b in FIG. 11B; movement 10812-b of contact 10810from location 10810-b in FIG. 11B to location 10810-c in FIG. 11C, FIG.11D, FIG. 11E or FIG. 11F; and/or liftoff of contact 10810 from location10810-c in FIG. 11G or FIG. 11H). Contact 10810 is detected at aposition on touch-sensitive surface 451 corresponding to an area ondisplay 450 occupied by control icon 10802 (e.g., contact 10810corresponds to a focus selector on the display, such as cursor 10806which is at or near a location of user interface object 10802). In someembodiments, movement of contact 10810 on touch-sensitive surface 451that corresponds to movement of focus selector (e.g., a cursor 10806) ondisplay 450 (e.g., as illustrated in FIGS. 11A-11F).

FIGS. 11A-11B illustrate an example of a beginning of a gesture wherethe device adjusts a size of images 10814, 10816, 10818, 10820, 10822,10824, 10826, 10828 and 10830 in accordance with movement 10812-a ofcontact 10810 that controls movement of cursor 10806 corresponding tomovement of a control icon 10802 in sizing bar 10804. In FIG. 11B, thedevice does not generate a tactile output corresponding to exceeding thepredefined adjustment limit (e.g., a size limit corresponding to the endof sizing bar 10804), because the predefined adjustment limit has notbeen exceeded. FIG. 11B illustrates an example where, in accordance witha determination that the adjustment of the parameter (e.g., size) wouldnot cause one or more predefined adjustment limit to be exceeded (e.g.,movement of control icon 10802 into a respective area of the displaythat is within a predefined adjustment limit indicated by the rightboundary of sizing bar 10804), the electronic device adjusts theparameter without generating a tactile output on the touch-sensitivesurface (e.g., the device increases the size of images 10814, 10816,10818, 10820, 10822, 10824, 10826, 10828 and 10830 displayed in userinterface 10808 to a size smaller than the predefined maximum size limitin accordance with the value of the parameter that corresponds to thecurrent location of control icon 10802 in the respective area of thedisplay). In contrast, FIGS. 11C-11F, described below, illustrateexamples where, in accordance with a determination that the adjustmentof the parameter (e.g., size) would cause one or more predefinedadjustment limit to be exceeded (e.g., where the movement of contact10810 on the touch-sensitive surface 451 corresponds to a sizeadjustment of the displayed pictures exceeding a predefined limitindicated by the right boundary of sizing bar 10804), tactile outputgenerators 167 generate tactile outputs 10832 on touch-sensitive surface451.

FIGS. 11B-11F illustrate various examples where the device detects acontinuation of a gesture including movement 10812-b of contact 10810that controls movement of cursor 10806 beyond an end of sizing bar10804. In FIG. 11C, in response to detecting the continuation of thegesture including movement 10812-b, the device continues to increase thesize of the images up to the predefined adjustment limit (e.g., a sizelimit) and moves control icon 10802 beyond an end of sizing bar 10804 inaccordance with movement of cursor 10806. In FIG. 11D, in response todetecting the continuation of the gesture including movement 10812-b,the device continues to increase the size of the images up to thepredefined adjustment limit (e.g., a size limit) and moves control icon10802 up to an end of sizing bar 10804 in accordance with movement ofcursor 10806. In FIG. 11E, in response to detecting the continuation ofthe gesture including movement 10812-b, the device cancels the increasein size of the images corresponding to the gesture and moves controlicon 10802 up to an end of sizing bar 10804 in accordance with movementof cursor 10806. In FIG. 11F, in response to detecting the continuationof the gesture including movement 10812-b, the device continues toincrease the size of the images beyond the predefined adjustment limit(e.g., a size limit) and moves control icon 10802 beyond an end ofsizing bar 10804 in accordance with movement of cursor 10806.

FIGS. 11C and 11F illustrate examples where the device detects movement10812-b of contact 10810 on touch-sensitive surface 451 that correspondsto movement of cursor 10806 and control icon 10802 past the rightboundary of sizing bar 10804 displayed on display 450 (e.g., movement ofcursor 10806 and control icon 10802 into a respective area of thedisplay that corresponds to an parameter adjustment exceeding apredefined adjustment limit for the displayed content).

FIGS. 11D-11E illustrate examples where the device detects movement10812-b of contact 10810 on touch-sensitive surface 451 that correspondsto movement of cursor 10806 past the right boundary of sizing bar 10804displayed on display 450 (e.g., movement of control icon 10802 into arespective area of the display that corresponds to a parameteradjustment exceeding a predefined adjustment limit for the displayedcontent) and movement of control icon 10802 to the right boundary ofsizing bar 10804 displayed on the display (e.g., movement of controlicon 10802 into a respective area of the display that corresponds to ana predefined adjustment limit for the displayed content, even though theextent of the movement 10812-b of contact 10810 corresponds to anadjustment of the parameter that exceeds the predefined adjustmentlimit).

FIGS. 11C-11F illustrate examples where, in accordance with adetermination that the adjustment of the parameter (e.g., size) wouldcause one or more predefined adjustment limit to be exceeded (e.g.,where the extent of the movement 10812-b of contact 10810 corresponds toa size adjustment of the displayed pictures exceeding a predefined limitindicated by the right boundary of sizing bar 10804), tactile outputgenerators 167 generate tactile outputs 10832 on touch-sensitive surface451 that indicate to the user that the gesture corresponds to anadjustment of the parameter that would exceed the predefined adjustmentlimit.

FIGS. 11C-11D illustrate examples where, in accordance with adetermination that the adjustment of the parameter would cause the oneor more predefined adjustment limits to be exceeded, the parameter isadjusted so that the predefined adjustment limit is reached (e.g., thesize of images 10814, 10816, 10818, 10820, 10822, 10824, 10826, 10828and 10830 displayed on user interface 10808 is increased to thepredefined maximum size limit).

FIG. 11E illustrates an example where, in accordance with adetermination that the adjustment of the parameter would cause the oneor more predefined adjustment limits to be exceeded, adjustment of theparameter is cancelled (e.g., the size of images 10814, 10816, 10818,10820, 10822, 10824, 10826, 10828 and 10830 displayed on user interface10808 is adjusted back to the size the images were displayed at prior todetecting the movement 10812-a and 10812-b of contact 10810).

FIG. 11F illustrates an example where, in accordance with adetermination that the adjustment of the parameter would cause the oneor more predefined adjustment limits to be exceeded, the parameter isadjusted accordingly such that the predefined adjustment limit isexceeded (e.g., the size of images 10814, 10816, 10818, 10820, 10822,10824, 10826, 10828 and 10830 displayed on user interface 10808 isincreased past the predefined maximum size limit).

FIGS. 11G-11H illustrate various examples where the device detectsliftoff of a contact 10810 used to perform one of the gestures describedabove with reference to FIGS. 11A-11F. FIG. 11G illustrates an examplewhere, in response to liftoff of contact 10810 in FIG. 11F, adjustmentof the parameter exceeding the predefined adjustment limit is notreversed (e.g., the size of images 10814, 10816, 10818, 10820, 10822,10824, 10826, 10828 and 10830 displayed on user interface 10808,increased past the predefined maximum size limit in FIG. 11F, ismaintained after liftoff of contact 10810). In this example, in responseto detecting liftoff of contact 10810, the device moves control icon10802 back to an end of scroll bar 10804. FIG. 11H illustrates anexample where, in response to liftoff of contact 10810 in FIG. 11F,adjustment of the parameter exceeding the predefined adjustment limit ispartially reversed to match the predefined adjustment limit (e.g., thesize of images 10814, 10816, 10818, 10820, 10822, 10824, 10826, 10828and 10830 displayed on user interface 10808, increased past thepredefined maximum size limit in FIG. 11F, is shrunk back down to thepredefined maximum size limit after liftoff of contact 10810). FIG. 11Gillustrates an example where, in response to liftoff of contact 10810 inFIG. 11F, display of control icon 10802 is adjusted to correspond to theright boundary of sizing bar 10804 (e.g., movement of control icon 10802back to the right boundary of sizing bar 10804).

FIGS. 11I-11K illustrate a contact 10840 and a gesture includingmovement 10842 of contact 10840 that are detected on touch-sensitivesurface 451 (e.g., movement 10842-a of contact 10840 from location10840-a in FIG. 11I to location 10840-b in FIG. 11J and/or movement10842-b of contact 10840 from location 10840-b in FIG. 11J to location10840-c in FIG. 11K). Contact 10840 is detected at a position ontouch-sensitive surface 451 corresponding to an area on display 450occupied by control 10836 (e.g., contact 10840 corresponds to a focusselector on the display, such as cursor 10806 which is at or near alocation of user interface object 10836). The gesture in FIGS. 11I-11Kincludes movement 10842 of contact 10840 on touch-sensitive surface 451that corresponds to movement of a focus selector (e.g., a cursor 10806)on display 450.

In some embodiments, as illustrated in FIGS. 11I-11K, the content is amedia clip (e.g., media clips 10844, 10846, 10848, 10850, 10852 and/or10854) that includes audio (e.g., audio 10834), the parameter is avolume level (e.g., volume level 10836), the predefined adjustmentlimits include a clipping limit (e.g., clipping limit 10838), and theclipping limit is exceeded when a maximum volume that occurs in thecontent is above the clipping limit. For example, as illustrated in FIG.11J, in response to detecting movement 10842-a of contact 10810corresponding to adjustment of a parameter that would not cause apredefined adjustment limit to be exceeded (e.g., increasing volumelevel 10836 such that the maximum volume of audio 10834 does not exceedvolume clipping limit 10838), the electronic device adjusts theparameter without generating a tactile output on the touch-sensitivesurface. In contrast, as illustrated in FIG. 11K, in response todetecting movement 10842-b of contact 10810 corresponding to adjustmentof the parameter that would cause the predefined adjustment limit to beexceeded (e.g., increasing volume level 10836 such that the maximumvolume of audio 10834 exceeds volume clipping limit 10838), tactileoutput generators 167 generate tactile outputs 10832 on touch-sensitivesurface 451.

FIGS. 11L-11N illustrate a contact 10860 and a gesture includingmovement 10862 of contact 10860 that are detected on touch-sensitivesurface 451 (e.g., movement 10862-a of contact 10860 from location10860-a in FIG. 11L to location 10860-b in FIG. 11M and/or movement10862-b of contact 10860 from location 10860-b in FIG. 11M to location10860-c in FIG. 11N). Contact 10860 is detected at a position ontouch-sensitive surface 451 corresponding to an area on display 450occupied by control 10858 (e.g., contact 10860 corresponds to a focusselector on the display, such as cursor 10864 which is at or near alocation of user interface object 10858). The gesture in FIGS. 11L-11Nincludes movement 10862 of contact 10860 on touch-sensitive surface 451that corresponds to movement of a focus selector (e.g., a cursor 10864)on display 450).

In some embodiments, as illustrated in FIGS. 11L-11N, the content is amedia clip (e.g., media clip 10856), the parameter is a cropping mask(e.g., cropping mask 10858), the predefined adjustment limits include atime-based content boundary (e.g., the right boundary of media clip10856), and the time-based content boundary is exceeded when thecropping mask extends beyond the time-based content boundary. Forexample, as illustrated in FIG. 11M, in response to detecting movement10862-a of contact 10860 corresponding to adjustment of a that would notcause a predefined adjustment limit to be exceeded (e.g., extendingcropping mask 10864 to, but not past, the right boundary of media clip10856), the electronic device adjusts the parameter (e.g., size ofcropping mask 10858) without generating a tactile output on thetouch-sensitive surface. In contrast, as illustrated in FIG. 11N, inresponse to detecting movement 10862-b of contact 10860 corresponding toadjustment of the parameter that would cause the predefined adjustmentlimit to be exceeded (e.g., extending cropping mask 10864 past the rightboundary of media clip 10856), tactile output generators 167 generatetactile outputs 10832 on touch-sensitive surface 451.

FIGS. 11O-11Q illustrate a contact 10870 and a gesture includingmovement 10872 of contact 10870 that are detected on touch-sensitivesurface 451 (e.g., movement 10872-a of contact 10870 from location10870-a in FIG. 11O to location 10870-b in FIG. 11P and/or movement10872-b of contact 10870 from location 10870-b in FIG. 11P to location10870-c in FIG. 11Q). Contact 10870 is detected at a position ontouch-sensitive surface 451 corresponding to an area on display 450occupied by control 10866 (e.g., contact 10870 corresponds to a focusselector on the display, such as cursor 10806 which is at or near alocation of user interface object 10866). The gesture in FIGS. 11O-11Qincludes movement 10872 of contact 10870 on touch-sensitive surface 451that corresponds to movement of a focus selector (e.g., a cursor 10806)on display 450.

In some embodiments, as illustrated in FIGS. 11O-11Q, the content is animage (e.g., image 10868), the parameter is a cropping mask (e.g.,cropping mask 10866), the predefined adjustment limits include a contentboundary (e.g., an outer edge of image 10868), and the content boundaryis exceeded when the cropping mask extends beyond the content boundary.For example, as illustrated in FIG. 11P, in response to detectingmovement 10872-a of contact 10870 corresponding to adjustment of aparameter that would not cause a predefined adjustment limits to beexceeded (e.g., extending cropping mask 10866 to, but not past, thelower and right borders of image 10868), the electronic device adjuststhe parameter (e.g., size of cropping mask 10866) without generating atactile output on the touch-sensitive surface. In contrast, asillustrated in FIG. 11Q, in response to detecting movement 10872-bcorresponding to adjustment of the parameter that would cause apredefined adjustment limits to be exceeded (e.g., extending croppingmask 10866 past the lower and right borders of image 10868), tactileoutput generators 167 generate tactile outputs 10832 on touch-sensitivesurface 451.

FIGS. 11R-11T illustrate a contact 10880 and a gesture includingmovement 10882 of contact 10880 that are detected on touch-sensitivesurface 451 (e.g., movement 10882-a of contact 10880 from location10880-a in FIG. 11R to location 10880-b in FIG. 11S and/or movement10882-b of contact 10880 from location 10880-b in FIG. 11S to location10880-c in FIG. 11T). Contact 10880 is detected at a position ontouch-sensitive surface 451 corresponding to an area on display 450occupied by control 10878 (e.g., contact 10880 corresponds to a focusselector on the display, such as cursor 10806 which is at or near alocation of user interface object 10878). The gesture in FIGS. 11R-11Tincludes movement 10882 of contact 10880 on touch-sensitive surface 451that corresponds to movement of a focus selector (e.g., a cursor 10806)on display 450.

In some embodiments, as illustrated in FIGS. 11R-11T, the content is ashape (e.g., square 10876), the parameter is a shape adjustmentparameter (e.g., roundness of the corners on square 10876), and thepredefined adjustment limits include a maximum allowable adjustment to arespective portion of a respective shape (e.g., maximum radius forrounding the corners of square 10876). For example, as illustrated inFIG. 11S, in response to detecting movement 10882-a corresponding toadjustment of a parameter that would not cause a predefined adjustmentlimits to be exceeded (e.g., rounding the corners of square 10876 usinga radius that matches, but does not exceed, a predetermined maximumradius), the electronic device adjusts the parameter (e.g., roundness ofthe corners of square 10876) without generating a tactile output on thetouch-sensitive surface. In contrast, as illustrated in FIG. 11T, inresponse to detecting movement 10882-b corresponding to adjustment ofthe parameter that would cause a predefined adjustment limits to beexceeded (e.g., rounding the corners of square 10876 using a radius thatexceeds a predetermined maximum radius), tactile output generators 167generate tactile outputs 10832 on touch-sensitive surface 451.

FIGS. 12A-12B are flow diagrams illustrating a method 10900 of providingfeedback when an action will result in the adjustment of a parameterbeyond a predefined limit in accordance with some embodiments. Themethod 10900 is performed at an electronic device (e.g., device 300,FIG. 3, or portable multifunction device 100, FIG. 1A) with a displayand a touch-sensitive surface. In some embodiments, the display is atouch screen display and the touch-sensitive surface is on the display.In some embodiments, the display is separate from the touch-sensitivesurface. Some operations in method 10900 are, optionally, combinedand/or the order of some operations is, optionally, changed.

As described below, the method 10900 provides an intuitive way toprovide feedback when an action will result in the adjustment of aparameter beyond a predefined limit. The method reduces the cognitiveburden on a user when detecting feedback when an action will result inthe adjustment of a parameter beyond a predefined limit, therebycreating a more efficient human-machine interface. For battery-operatedelectronic devices, enabling a user to detect feedback when an actionwill result in the adjustment of a parameter beyond a predefined limitfaster and more efficiently conserves power and increases the timebetween battery charges.

In some embodiments, the device displays (10902), on a display (e.g.,display 450 in FIGS. 11A-11T), a control (e.g., a resizing controlincluding sizing bar 10804 and control icon 10802 in FIGS. 11A-11H,control 10836 in FIGS. 11I-11K, control 10858 in FIGS. 11L-11N, control10866 in FIGS. 11O-11Q, or control 10878 in FIGS. 11R-11T) forcontrolling a parameter associated with respective content (e.g., thesize of images 10814, 10816, 10818, 10820, 10822, 10824, 10826, 10828and 10830 in FIGS. 11A-11H, the volume level of audio 10834 in FIGS.11I-11K, cropping mask 10858 applied to media clip 10856 in FIGS.11L-11N, cropping mask 10866 applied to image 10868 in FIGS. 11O-11Q, orthe roundness of the corners of square 10876 in FIGS. 11R-11T).

In some embodiments, while the device displays the control, the devicedetects (10904) a gesture (e.g., movement 10812 of contact 10810 inFIGS. 11A-11H, movement 10842 of contact 10840 in FIGS. 11I-11K,movement 10862 of contact 10860 in FIGS. 11L-11N, movement 10872 ofcontact 10870 in FIGS. 11O-11Q, or movement 10882 of contact 10880 inFIGS. 11R-11T) on a touch-sensitive surface (e.g., touch-sensitivesurface 451) for adjusting the parameter.

In some embodiments, in response (10906) to detecting the gesture: thedevice determines (10908) an adjustment of the parameter thatcorresponds to an extent of the gesture (e.g., an extent of lateralmovement, an extent of rotation, or an extent of increase/decrease inintensity of a contact detected on the touch-sensitive surface).

In response (10906) to detecting the gesture: in accordance with adetermination that the adjustment of the parameter would cause one ormore predefined adjustment limits to be exceeded, the device generates(10910) a respective tactile output (e.g., tactile outputs 10832 inFIGS. 11C-11F, FIG. 11K, FIG. 11N, FIG. 11Q or FIG. 11T) on thetouch-sensitive surface (e.g., touch-sensitive surface 451).

In some embodiments, in response (10906) to detecting the gesture: inaccordance with a determination that the adjustment of the parameterwould cause one or more predefined adjustment limits to be exceeded, thedevice forgoes (10912) adjusting the parameter (e.g., adjustment of theparameter is cancelled if the requested adjustment exceeds theadjustment limits for the parameter). For example, as illustrated inFIGS. 11C and 11E, where a gesture including movement 10812-b of contact10810 that corresponds to a size adjustment of images 10814, 10816,10818, 10820, 10822, 10824, 10826, 10828 and 10830 that would exceed thepredefined adjustment limit corresponding to the right boundary ofsizing bar 10804, the images are displayed at a magnificationcorresponding to the size of the images prior to the gesture, as shownin FIG. 11E.

In some embodiments, in response (10906) to detecting the gesture: inaccordance with a determination that the adjustment of the parameterwould cause the one or more predefined adjustment limits to be exceeded,the device adjusts (10914) the parameter so that the predefinedadjustment limit is reached (e.g., the extent of the adjustment of theparameter is limited by the predetermined adjustment limit). Forexample, as illustrated in FIGS. 11C-11D, where the gesture includingmovement 10812-b of contact 10810 corresponds to a size adjustment ofimages 10814, 10816, 10818, 10820, 10822, 10824, 10826, 10828 and 10830that would exceed the predefined adjustment limit corresponding to theright boundary of sizing bar 10804, the images are displayed at amagnification corresponding to the predefined maximum size adjustmentlimit, as shown in FIG. 11D. In some embodiments, the parameter isadjusted to a respective predefined adjustment limit, but the respectivepredefined adjustment limit is not exceeded (e.g., a thumb on a slideris adjusted to the end of the slider, but no further).

In some embodiments, in response (10906) to detecting the gesture: inaccordance with a determination that the adjustment of the parameterwould cause the one or more predefined adjustment limits to be exceeded,the device performs (10916) the adjustment of the parameter (e.g., theextent of the adjustment of the parameter is not limited by thepredetermined adjustment limit). For example, as illustrated in FIG.11F, where the gesture including movement 10812-b of contact 10810corresponds to a size adjustment of images 10814, 10816, 10818, 10820,10822, 10824, 10826, 10828 and 10830 that would exceed the predefinedadjustment limit corresponding to the right boundary of sizing bar10804, the images are displayed at a magnification corresponding to thesize exceeding the predefined maximum size adjustment limit (e.g., evenif adjusting the parameter in this way causes the one or more predefinedadjustment limits to be exceeded), as shown in FIG. 11F. For example, auser is allowed to extend a cropping mask outside of a canvas (e.g., asshown in FIG. 11Q) or beyond an end of a media clip (e.g., as shown inFIG. 11N), but the device warns the user that cropping while thecropping mask is outside of the canvas or beyond the end of the mediaclip will add blank content to the respective content.

In some embodiments, the content is (10918) a media clip (e.g., mediaclips 10844, 10846, 10848, 10850, 10852 and/or 10854 in FIGS. 11I-11K)that includes audio (e.g., audio 10834 in FIGS. 11I-11K), the parameteris a volume level (e.g., volume level 10836 in FIGS. 11I-11K), thepredefined adjustment limits include a clipping limit (e.g., clippinglimit 10838 in FIGS. 11I-11K), and the clipping limit is exceeded when amaximum volume that occurs in the content is above the clipping limit(e.g., when the tallest peaks of audio 10834 exceed clipping limit 10838in FIG. 11K). In some embodiments, the clipping limit is an analogclipping limit (e.g., the sound is limited by the physical/electricalcharacteristics of an amplifier such that sounds above the clippinglimit would push an amplifier to create a signal with more power thanits power supply can produce and thus the amplifier amplifies the signalonly up to its maximum capacity). In some embodiments, the clippinglimit is a digital clipping limit (e.g., the signal is restricted by thepredetermined range digital representations of the sound and a soundwith amplitude above the range of chosen digital representations will berepresented as a maximum digital representation).

In some embodiments, the content is (10920) a media clip (e.g., mediaclip 10856 in FIGS. 11L-11N), the parameter is a cropping mask (e.g.,cropping mask 10858 in FIGS. 11L-11N), the predefined adjustment limitsinclude a time-based content boundary (e.g., the right boundary of mediaclip 10856 in FIGS. 11L-11N), and the time-based content boundary isexceeded when the cropping mask extends beyond the time-based contentboundary (e.g., the user tries to perform an operation that correspondsto cropping the media clip before the beginning time of the media clip,or after the end time of the media clip). For example, as illustrated inFIG. 11N, the time-based content boundary is exceeded when cropping mask10858 extends past the right time-based content boundary of media clip10856.

In some embodiments, the content is (10922) an image (e.g., image 10868in FIGS. 11O-11Q), the parameter is a cropping mask (e.g., cropping mask10866 in FIGS. 11O-11Q), the predefined adjustment limits include acontent boundary (e.g., an outer edge of image 10868 in FIGS. 11O-11Q),and the content boundary is exceeded when the cropping mask extendsbeyond the content boundary (e.g., beyond the border of an image). Forexample, as illustrated in FIG. 11Q, the content boundary is exceededwhen cropping mask 10866 extends beyond the lower and right borders ofimage 10868.

In some embodiments, the content is (10924) a shape (e.g., square 10876in FIG. 11R), the parameter is a shape adjustment parameter (e.g., aparameter corresponding to the roundness of the corners on square 10876in FIGS. 11R-11T), and the predefined adjustment limits include amaximum allowable adjustment to a respective portion of a respectiveshape (e.g., maximum radius for rounding the corners of square 10876 inFIGS. 11R-11T). In some embodiments, the shape adjustment parameter is,for example, roundness of a shape corner, opacity or line width. In someembodiments, the predefined adjustment limit is, for example, a maximumradius for rounding a corner, a minimum/maximum opacity, or aminimum/maximum line width.

In response (10906) to detecting the gesture: in accordance with adetermination that the adjustment of the parameter would not cause theone or more predefined adjustment limits to be exceeded, the deviceperforms (10926) the adjustment of the parameter without generating therespective tactile output (e.g., a tactile output corresponding toexceeding the predefined adjustment limit) on the touch-sensitivesurface (e.g., touch-sensitive surface 451). For example, as illustratedin FIGS. 11B, 11J, 11M, 11P and 11S, where a gesture corresponds to anadjustment of the parameter that does not exceed a predefined adjustmentlimit, the adjustment is performed and no tactile output is generated onthe touch-sensitive surface.

It should be understood that the particular order in which theoperations in FIGS. 12A-12B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,those listed in paragraph [0058]) are also applicable in an analogousmanner to method 10900 described above with respect to FIGS. 12A-12B.For example, the contacts, gestures, user interface objects, tactilesensations and focus selectors described above with reference to method10900 optionally have one or more of the characteristics of thecontacts, gestures, user interface objects, tactile sensations and focusselectors described herein with reference to other methods describedherein. For brevity, these details are not repeated here.

In accordance with some embodiments, FIG. 13 shows a functional blockdiagram of an electronic device 11000 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 13 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 13, an electronic device 11000 includes a display unit11002 configured to display a control for controlling a parameterassociated with respective content, a touch-sensitive surface unit 11004configured to receive user contacts, optionally one or more sensor units11006 configured to detect intensity of contacts with thetouch-sensitive surface unit 11004; and a processing unit 11008 coupledto the display unit 11002, the touch-sensitive surface unit 11004 andoptionally the one or more sensor units 11006. In some embodiments, theprocessing unit 11008 includes a display enabling unit 11010, adetecting unit 11012, a determining unit 11014, a generating unit 11016,and an adjusting unit 11018.

In some embodiments, the processing unit 11008 is configured to enabledisplay (e.g., with the display enabling unit 11010) of a control forcontrolling a parameter associated with respective content. In someembodiments, the processing unit 11008 is further configured to detect agesture on the touch-sensitive surface unit 11004 for adjusting theparameter (e.g., with the detecting unit 11012); and in response todetecting the gesture: the processing unit 11008 is configured determinean adjustment of the parameter that corresponds to an extent of thegesture (e.g., with the determining unit 11014); in accordance with adetermination that the adjustment of the parameter would cause one ormore predefined adjustment limits to be exceeded, the processing unit11008 is configured to generate a respective tactile output on thetouch-sensitive surface unit (e.g., with the generating unit 11016); andin accordance with a determination that the adjustment of the parameterwould not cause the one or more predefined adjustment limits to beexceeded, the processing unit 11008 is configured to perform theadjustment of the parameter (e.g., with the adjusting unit 11018)without generating the respective tactile output on the touch-sensitivesurface unit 11004.

In some embodiments, the processing unit 11008 is further configured to,in accordance with a determination that the adjustment of the parameterwould cause the one or more predefined adjustment limits to be exceeded,forgo adjusting the parameter (e.g., with the adjusting unit 11018).

In some embodiments, the processing unit 11008 is further configured to,in accordance with a determination that the adjustment of the parameterwould cause the one or more predefined adjustment limits to be exceeded,adjust the parameter so that the predefined adjustment limit is reached(e.g., with the adjusting unit 11018).

In some embodiments, the processing unit 11008 is further configured to,in accordance with a determination that the adjustment of the parameterwould cause the one or more predefined adjustment limits to be exceeded,perform the adjustment of the parameter (e.g., with the adjusting unit11018).

In some embodiments, the content is a media clip that includes audio,the parameter is a volume level, the predefined adjustment limitsinclude a clipping limit and the clipping limit is exceeded when amaximum volume that occurs in the content is above the clipping limit.

In some embodiments, the content is a media clip, the parameter is acropping mask, the predefined adjustment limits include a time-basedcontent boundary and the time-based content boundary is exceeded whenthe cropping mask extends beyond the time-based content boundary.

In some embodiments, the content is an image, the parameter is acropping mask, the predefined adjustment limits include a contentboundary and the content boundary is exceeded when the cropping maskextends beyond the content boundary.

In some embodiments, the content is a shape, the parameter is a shapeadjustment parameter and the predefined adjustment limits include amaximum allowable adjustment to a respective portion of a respectiveshape.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 12A-12B are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.13. For example, detection operation 10904 and determination operations10908, 10910, 10912, 10914, 10916 and 10926 are, optionally, implementedby event sorter 170, event recognizer 180, and event handler 190. Eventmonitor 171 in event sorter 170 detects a contact on touch-sensitivedisplay 112, and event dispatcher module 174 delivers the eventinformation to application 136-1. A respective event recognizer 180 ofapplication 136-1 compares the event information to respective eventdefinitions 186, and determines whether a first contact at a firstlocation on the touch-sensitive surface corresponds to a predefinedevent or sub-event, such as selection of an object on a user interface,adjustment of a parameter associated with respective content, orgeneration of a tactile output (e.g., corresponding to a determinationthat an adjustment of a parameter would cause one or more predefinedadjustment limits to be exceeded). When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally utilizes or calls data updater 176 or objectupdater 177 to update the application internal state 192. In someembodiments, event handler 190 accesses a respective GUI updater 178 toupdate what is displayed by the application. Similarly, it would beclear to a person having ordinary skill in the art how other processescan be implemented based on the components depicted in FIGS. 1A-1B.

Providing Tactile Feedback Corresponding to a Clock

Many electronic devices have graphical user interfaces that include arepresentation of a clock. For example, many cellular phones, laptops,and tablets have a representation of a clock prominently displayed onthe graphical user interface. There is often a need to provide efficientand convenient ways for users to receive feedback corresponding to theclock. The embodiments below improve on existing methods by generatingtactile outputs for the user that correspond to the clock (e.g., a ‘ticktock’ pattern of tactile outputs) indicating that a focus selector isover the representation of the clock and, optionally, providing anindication of the rate at which time is passing.

FIGS. 14A-14J illustrate exemplary user interfaces for providing tactilefeedback corresponding to a clock in accordance with some embodiments.The user interfaces in these figures are used to illustrate theprocesses described below, including the processes described below withreference to FIGS. 15A-15B.

FIG. 14A illustrates an example of a user interface that includes arepresentation of a clock. User interface 11100 is displayed on display450 of a device (e.g., device 300) and is responsive to contacts (e.g.,a finger contact) on touch-sensitive surface 451. User interface 11100includes representation 11102 of a clock. FIG. 14A further illustratescontact 11106 on touch-sensitive surface 451 and, per some embodiments,a displayed representation of focus selector (e.g., a cursor 11104), atposition 11104-a, corresponding to contact 11106.

In some embodiments, the device is portable multifunction device 100,the display is touch-sensitive display system 112, and thetouch-sensitive surface includes tactile output generators 167 on thedisplay (FIG. 1A). For convenience of explanation, the embodimentsdescribed with reference to FIGS. 14A-14J and FIGS. 15A-15B will bediscussed with reference to display 450 and a separate touch-sensitivesurface 451, however analogous operations are, optionally, performed ona device with a touch-sensitive display system 112 in response todetecting the contacts described in FIGS. 14A-14J on the touch-sensitivedisplay system 112 while displaying the user interfaces shown in FIGS.14A-14J on the touch-sensitive display system 112; in such embodiments,the focus selector is, optionally: a respective contact, arepresentative point corresponding to a contact (e.g., a centroid of arespective contact or a point associated with a respective contact), ora centroid of two or more contacts detected on the touch-sensitivedisplay system 112, in place of cursor 11104.

FIGS. 14A and 14B illustrate an example of detecting a focus selectorover a representation of a clock. In this example, contact 11106 andmovement 11108 of contact 11106 are detected on touch-sensitive surface451. Movement of a focus selector (e.g., cursor 11104), corresponding tomovement 11108, causes the focus selector (e.g., cursor 11104) to moveto from position 11104-a in FIG. 14A that is not over representation11102 of a clock to position 11104-b in FIG. 14B that is over therepresentation 11102 of the clock and the device starts to providetactile feedback 11110 (e.g., generating tactile outputs correspondingto a tick tock sensation) on touch-sensitive surface 451.

FIG. 14C illustrates an example of continuing to provide tactilefeedback while detecting a focus selector over a representation of aclock. In this example, cursor 11104 at position 11104-b is overrepresentation 11102 of a clock and tactile feedback 11110 continues tobe provided on touch-sensitive surface 451. As discussed below withreference to FIG. 14B, tactile feedback 11110 includes a regular patternof tactile outputs on touch-sensitive surface 451.

FIGS. 14C and 14D illustrate an example of movement of a focus selectorthat maintains the focus selector over a representation of a clock. Inthis example, a focus selector (e.g., cursor 11104) is initially atposition 11104-b as shown in FIG. 14C. As shown in FIG. 14D, contact11112 and movement 11114 are detected on touch-sensitive surface 451 andthe corresponding movement of cursor 11104 causes cursor 11104 to moveto position 11104-c. Since cursor 11104 is over representation 11102 ofa clock when at position 11104-c, tactile feedback 11110 is provided ontouch-sensitive surface 451. In some embodiments, the period of theregular pattern of tactile feedback is not based on movement of a focusselector (e.g., cursor 11104) that maintains the focus selector (e.g.,cursor 11104) over representation 11102 of the clock. For example, insome embodiments, while the cursor 11104 remains over representation11102 of the clock, the period of the regular pattern of tactilefeedback is independent of movement of cursor 11104.

FIGS. 14D and 14E illustrate an example of movement of a focus selectoraway from a representation of a clock and therefore ceasing to providetactile feedback corresponding to the clock. In this example, cursor11104 is initially at position 11104-c as shown in FIG. 14D. As shown inFIG. 14E, contact 11116 and movement 11118 are detected ontouch-sensitive surface 451 and the corresponding movement of cursor11104 causes cursor 11104 to move to position 11104-d. Since cursor11104 is no longer over representation 11102 of a clock when at position11104-d, tactile feedback is no longer provided on touch-sensitivesurface 451.

FIGS. 14F-14H illustrate example waveforms of movement profiles forgenerating the tactile feedback. FIG. 14F illustrates a trianglewaveform with period 11130-1. FIG. 14G illustrates a square waveformwith period 11130-2 and FIG. 14H illustrates a sawtooth waveform withperiod 11130-3. In some embodiments, one of the movement profilesillustrated in FIGS. 14F-14H will be utilized when generating tactilefeedback 11110 corresponding to a clock, as discussed above. In theseexamples, since the regular pattern comprises repetition of singlewaveform, the period of the regular pattern is the same as the period ofthe individual waveform in the regular pattern. In some embodiments, theperiod (e.g., from peak to peak or leading edge to leading edge) of theregular pattern is 1 second.

Per some embodiments, FIGS. 14I-14J illustrate example waveforms ofmovement profiles that include an alternating sequence of outputs. Insome embodiments, tactile feedback 11110 includes an alternatingsequence of tactile outputs that have different output characteristics.FIG. 14I illustrates an alternating sequence of square waves withapproximately the same period and with different amplitudes 11132-1 and11132-2. FIG. 14J illustrates an alternating sequence of square andsawtooth waves with approximately the same period and amplitudes. Insome embodiments, the period (e.g., from peak to peak or leading edge toleading edge of outputs with the same movement profile) of the regularpattern is 2 seconds, so that the time between two successive outputs is1 second or approximately 1 second (e.g., the time between a tick and atock is 1 second and the time between a tock and a tick is 1 second).

In some embodiments, the tactile feedback 11110 includes other regularpatterns of tactile outputs on touch-sensitive surface 451 than the onesshown in FIGS. 14F-14J. For example, the regular pattern can be asequence of component waveforms, having a sequence of length L (where Lis an integer greater than 0), that is repeatedly generated. In someembodiments, at least one component in the sequence of componentwaveforms is distinct from at least one of the other components in atleast one respect (e.g., amplitude, period and/or shape). In someembodiments, each component waveform in the sequence of componentwaveforms is distinct from the other components in at least one respect(e.g., amplitude, period and/or shape), while in other embodiments, somecomponents (“repeated components”) in the sequence are the same (e.g.,every Nth component in the sequence of L components is the same), whileother components are different from the repeated components. In theseembodiments, the period of the regular pattern is the period to generatethe sequence of component waveforms (i.e., from the start time of afirst instance of the regular pattern until the start time of a nextinstance of the regular pattern).

FIGS. 15A-15B are flow diagrams illustrating a method 11200 of providingtactile feedback corresponding to a clock in accordance with someembodiments. Method 11200 is performed at an electronic device (e.g.,device 300, FIG. 3, or portable multifunction device 100, FIG. 1A) witha display and a touch-sensitive surface. In some embodiments, thedisplay is a touch screen display and the touch-sensitive surface is onthe display. In some embodiments, the display is separate from thetouch-sensitive surface. Some operations in method 11200 are,optionally, combined and/or the order of some operations is, optionally,changed.

As described below, the method 11200 provides an intuitive way toprovide tactile feedback corresponding to a clock. The method reducesthe cognitive burden on a user when displaying a clock, thereby creatinga more efficient human-machine interface. For battery-operatedelectronic devices, enabling a user to interact with a clock faster andmore efficiently conserves power and increases the time between batterycharges.

The device displays (11202) a representation of a clock. FIG. 14A, forexample, shows representation 11102 of a clock, displayed in graphicaluser interface 11100. In some embodiments, prior to detecting the focusselector over the representation of the clock, the device displays(11204) the representation of the clock without providing the tactilefeedback that corresponds to the clock on the touch-sensitive surface(e.g., a tick-tock output corresponding to the clock is not generatedprior to the focus selector moving over the clock). In FIG. 14A, forexample, cursor 11104 is at position 11104-a and is not overrepresentation 11102 of the clock and therefore tactile feedback 11110is not generated by the device.

While displaying the representation of the clock, the device detects(11206) movement of a focus selector over the representation of theclock. As shown in FIG. 14B, for example, cursor 11104 moves to position11104-b over representation 11102 of the clock from a position 11104-athat was not over the representation of the clock. While detecting thefocus selector over the representation of the clock, the device provides(11208) tactile feedback that corresponds to the clock, where thetactile feedback includes a regular pattern of tactile outputs on thetouch-sensitive surface. For example, FIG. 14C shows cursor 11104 atposition 11104-b over representation 11102 of the clock and tactilefeedback 11110 provided on touch-sensitive surface 451, where thetactile feedback includes a regular pattern of tactile outputs, such asthose described above with reference to FIGS. 14F-14J.

In some embodiments, the tactile outputs in the regular pattern oftactile outputs are generated at evenly spaced intervals (11210). Forexample, in some embodiments, the regular pattern of tactile outputswill have a period of one second. In some other embodiments the regularpattern of tactile outputs will have a period of 0.5 seconds, 2 seconds,or other length of time between 0.25 seconds and ten seconds.

In some embodiments, the regular pattern of tactile outputs on thetouch-sensitive surface includes one of the regular patterns describedabove with reference to FIGS. 14F-14J. For example, in some embodiments,the regular pattern of tactile outputs on the touch-sensitive surfaceincludes an alternating sequence of tactile outputs that have differentoutput characteristics (11212). For example, in some embodiments, thepattern of tactile outputs on the touch-sensitive surface will generatea tick-tock sensation where “tick” tactile outputs and “tock” tactileoutputs are selected so as to produce “tick” sensations that correspondto “tick” tactile outputs feel different to a user than “tock”sensations that correspond to “tock” tactile outputs. FIGS. 14I-14J, forexample, illustrate example waveforms of movement profiles that includean alternating sequence of outputs.

In some embodiments, the alternating sequence of tactile outputsincludes a first type of tactile output alternating with a second typeof tactile output (11214) with a different amplitude. For example, insome embodiments, various combinations of the waveforms of movementprofiles illustrated in FIGS. 14F-14H would be utilized. In someembodiments, the first type of tactile output is generated by movementof the touch-sensitive surface that includes a first dominant movementcomponent (e.g., movement corresponding to an initial impulse of thefirst tactile output, ignoring any unintended resonance). In someembodiments, the second type of tactile output is generated by movementof the touch-sensitive surface that includes a second dominant movementcomponent (e.g., movement corresponding to an initial impulse of thesecond tactile output, ignoring any unintended resonance). In someembodiments, the first dominant movement component and the seconddominant movement component have a same movement profile (e.g., samewaveform shape such as square, sine, squine, sawtooth or triangle;and/or approximately the same width/period) and different amplitudes.FIG. 14I, for example, illustrates an example waveform of movementprofiles with alternating square waves having different amplitudes11132-1 and 11132-2.

In some embodiments, the alternating sequence of tactile outputsincludes a first type of tactile output alternating with a second typeof tactile output (11216) with a different movement profile. Forexample, in some embodiments, various combinations of the waveforms ofmovement profiles illustrated in FIGS. 14F-14H would be utilized. Insome embodiments, the first type of tactile output is generated bymovement of the touch-sensitive surface that includes a first dominantmovement component (e.g., movement corresponding to an initial impulseof the first tactile output, ignoring any unintended resonance). In someembodiments, the second type of tactile output is generated by movementof the touch-sensitive surface that includes a second dominant movementcomponent (e.g., movement corresponding to an initial impulse of thesecond tactile output, ignoring any unintended resonance). In someembodiments, the first dominant movement component and the seconddominant movement component have different movement profiles (e.g.,different waveform shapes such as square, sine, squine, sawtooth ortriangle; and/or different width/period) and a same amplitude. FIG. 14J,for example, illustrates an example waveform of movement profiles withalternating square waves and sawtooth waves with the same amplitude.

In some embodiments, while detecting (11218) the focus selector over therepresentation of the clock, the device detects (11220) movement of thefocus selector that maintains the focus selector over the representationof the clock and in response to detecting movement of the focusselector, the device continues to provide (11222) the tactile feedbackcorresponding to the clock without changing a period of the regularpattern. In some embodiments, the period of the regular pattern oftactile outputs is not changed based on movement of the focus selectorwhile over the representation of the clock. For instance, per theseembodiments, the period of tactile feedback 11110 would not change whencursor 11104 moves from position 11104-b in FIG. 14C to position 11104-cin FIG. 14D.

While providing the tactile feedback, the device detects (11224)movement of the focus selector away from the representation of theclock. FIG. 14E, for example, shows cursor 11104 moving to position11104-d away from representation 11102 of the clock. In response todetecting movement of the focus selector away from the representation ofthe clock, the device ceases (11226) to provide the tactile feedbackcorresponding to the clock. For example, this is illustrated in FIGS.14D-14E. In FIG. 14D, cursor 11104 is at position 11104-b overrepresentation 11102 of the clock and the device provides tactilefeedback 11110. However, in FIG. 14E, cursor 11104 moves to position11104-d away from representation 11102 of the clock and the deviceceases to generate tactile feedback 11110.

It should be understood that the particular order in which theoperations in FIGS. 15A-15B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein are alsoapplicable in an analogous manner to method 11200 described above withrespect to FIGS. 15A-15B. For example, the contacts, focus selectors,and tactile feedback (e.g., tactile outputs) described above withreference to method 11200 optionally has one or more of thecharacteristics of contacts, focus selectors, and tactile feedback(e.g., tactile outputs) described herein with reference to other methodsdescribed herein. For brevity, these details are not repeated here.

In accordance with some embodiments, FIG. 16 shows a functional blockdiagram of an electronic device 11300 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 16 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 16, an electronic device 11300 includes a display unit11302 configured to display a representation of a clock; atouch-sensitive surface unit 11304; and a processing unit 11306 coupledto the display unit 11302 and the touch-sensitive surface unit 11304. Insome embodiments, the processing unit includes detecting unit 11308,display enabling unit 11310, feedback unit 11312, and generating unit11314.

The processing unit 11306 is configured to: detect movement of a focusselector over the representation of the clock (e.g., with the detectingunit 11308), while detecting the focus selector over the representationof the clock, provide tactile feedback (e.g., with the feedback unit11312) that corresponds to the clock, where the tactile feedbackincludes a regular pattern of tactile outputs on the touch-sensitivesurface unit. The processing unit is further configured to, whileproviding the tactile feedback, detect movement of the focus selectoraway from the representation of the clock (e.g., with the detecting unit11308), and in response to detecting movement of the focus selector awayfrom the representation of the clock, cease to provide the tactilefeedback (e.g., with the feedback unit 11312) corresponding to theclock.

In some embodiments, the tactile outputs in the regular pattern oftactile outputs are generated at evenly spaced intervals (e.g., with thegenerating unit 11314).

In some embodiments, the processing unit 11306 is further configured to,while detecting the focus selector over the representation of the clock(e.g., with detecting unit 11308), detect movement of the focus selectorthat maintains the focus selector over the representation of the clock(e.g., with the detecting unit 11308), and in response to detectingmovement of the focus selector, continue to provide the tactile feedbackcorresponding to the clock (e.g., with feedback unit 11312) withoutchanging a period of the regular pattern.

In some embodiments, the processing unit 11306 is further configured to,prior to detecting the focus selector over the representation of theclock, display the representation of the clock on the display unit(e.g., with the display enabling unit 11310) without providing thetactile feedback that corresponds to the clock on the touch-sensitivesurface unit.

In some embodiments, the regular pattern of tactile outputs on thetouch-sensitive surface unit includes an alternating sequence of tactileoutputs that have different output characteristics.

In some embodiments, the alternating sequence of tactile outputsincludes a first type of tactile output alternating with a second typeof tactile output, the first type of tactile output is generated bymovement of the touch-sensitive surface unit (e.g., with the generatingunit 11314) that includes a first dominant movement component, thesecond type of tactile output is generated by movement of thetouch-sensitive surface unit (e.g., with the generating unit 11314) thatincludes a second dominant movement component, and the first dominantmovement component and the second dominant movement component have asame movement profile and different amplitudes.

In some embodiments, the alternating sequence of tactile outputsincludes a first type of tactile output alternating with a second typeof tactile output, the first type of tactile output is generated bymovement of the touch-sensitive surface unit (e.g., with the generatingunit 11314) that includes a first dominant movement component, thesecond type of tactile output is generated by movement of thetouch-sensitive surface unit (e.g., with the generating unit 11314) thatincludes a second dominant movement component, and the first dominantmovement component and the second dominant movement component havedifferent movement profiles and a same amplitude.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 15A-15B are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.16. For example, detection operations 11206, 11220 and 11224 and tactilefeedback operation 11208 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface corresponds to a predefined event or sub-event,such as selection of an object on a user interface. When a respectivepredefined event or sub-event is detected, event recognizer 180activates an event handler 190 associated with the detection of theevent or sub-event. Event handler 190 optionally utilizes or calls dataupdater 176 or object updater 177 to update the application internalstate 192. In some embodiments, event handler 190 accesses a respectiveGUI updater 178 to update what is displayed by the application.Similarly, it would be clear to a person having ordinary skill in theart how other processes can be implemented based on the componentsdepicted in FIGS. 1A-1B.

Providing Tactile Feedback Corresponding to Beats of a Piece of Music

Many electronic devices have graphical user interfaces that displayapplication windows showing representations of a piece of music (e.g., agraphical representation of a piece of cover art for an album of thepiece of music, a region indicating that a piece of music is beingcurrently being played or notes of a piece of music in a graphicalrepresentation of a music score corresponding to a piece of music). Forexample, a media player application window (e.g., an audio or videoplayer) optionally includes a display region that provides informationon a selected piece of music (e.g., the name, composer, artist,associated album, recording date, publisher and/or length of the pieceof music). Likewise, a composing application window optionally displaysan interactive representation of the musical score of a piece of musicbeing composed, allowing the user to manipulate the piece of music byadding, removing or changing notes displayed in the score. Given thecomplexity of user interface environment that includes applicationwindows corresponding to applications having both audio and visualcomponents (e.g., music playback, music composition, video playback orvideo composition applications), there is a need to provide feedbackthat enables the user to more efficiently and conveniently navigatethrough the user interface environment.

The embodiments described below provide improved methods and userinterfaces for generating feedback to a user navigating a complex userinterface environment. More specifically, these methods and userinterfaces provide feedback that corresponds to beats of a piece ofmusic represented on a display. The tactile feedback provides the userwith a sense of the beat of the piece of music. In this fashion, themethods and user interfaces provided below allow the user to moreefficiently and conveniently achieve an understanding of the beat of themusic, as well as a greater understanding of the piece of music as awhole, by providing tactile feedback, instead of or in addition toaudible and/or visual feedback. For example, in some embodiments, a usersearching for a piece of music with a specific beat (e.g., to accompanya visual display or to listen to while running) moves a focus selectorover a representation of the piece of music on the display, a receivestactile feedback corresponding to the beat of the music, without havingto listen to the music. Likewise, in some embodiments, a user composinga piece of music moves a focus selector over a representation of themusical piece they are composition and receives tactile feedbackcorresponding to the beat of the music, expediting the compositionprocess.

Some methods for sensing the beat of a piece of music rely on the userlistening to the piece of music and picking-up the beat themselves.Other methods for sensing the beat of a piece of music rely on the userdetecting a visual cue (e.g., a flash or pulse on a display)corresponding to the beat of the music. However, there are manysituations (e.g., at work, in a theatre and in various socialsituations) where the volume of an electronic device will be lowered ormuted, rendering audible cues ineffective. Advantageously, the methodsand user interfaces described below augment or replace audible byproviding tactile feedback indicating that a user interface object hasbeen selected and/or activated.

FIGS. 17A-17L illustrate exemplary user interfaces for providingfeedback that corresponds to beats of a piece of music in accordancewith some embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIGS. 18A-18B. FIGS. 17A-17B, 17G-17I and 17K-17L include intensitydiagrams that show the current intensity of the contact on thetouch-sensitive surface relative to a plurality of intensity thresholdsincluding a contact detection intensity threshold (e.g., “IT₀”) and alight press intensity threshold (e.g., “IT_(L)”). In some embodiments,operations similar to those described below with reference to IT_(L) areperformed with reference to a different intensity threshold (e.g.,“IT_(D)”). In some embodiments, the operations described below are notdependent on an intensity of the contact. FIGS. 17C, 17E-17G and 17Jinclude musical scores and waveform diagrams that show the amplitude(e.g., a high amplitude “A_(H)” or low amplitude “A_(L)”) and shape(e.g., square or sawtooth) of the waveform corresponding to tactileoutput generated on the touch-sensitive surface in response to a tactileoutput generating event (e.g., selection or playback of a beat in apiece of music). These musical scores and waveform diagrams aretypically not part of the displayed user interface, but are provided toaid in the interpretation of the figures.

FIG. 17A illustrates exemplary user interface 11408 displaying one ormore user interface objects, for example, user interface 11408 displaysmedia player window 11402 that includes representations of a piece ofmusic (e.g., graphical representations 11406-1, 11406-2, and 11406-3 ofa piece of cover art for a musical album) and cursor 11404 (e.g., adisplayed representation of a focus selector). In FIG. 17A, userinterface 11408 is displayed on display 450 of an electronic device thatalso includes touch-sensitive surface 451 and one or more sensors fordetecting intensity of contacts with touch-sensitive surface. In someembodiments, touch-sensitive surface 451 is a touch screen display thatis optionally display 450 or a separate display.

In some embodiments, the device is an electronic device with a separatedisplay (e.g., display 450) and a separate touch-sensitive surface(e.g., touch-sensitive surface 451). In some embodiments, the device isportable multifunction device 100, the display is touch-sensitivedisplay system 112, and the touch-sensitive surface includes tactileoutput generators 167 on the display (FIG. 1A). For convenience ofexplanation, the embodiments described with reference to FIGS. 17A-17Oand 18A-18B will be discussed with reference to display 450 and aseparate touch-sensitive surface 451, however analogous operations are,optionally, performed on a device with a touch-sensitive display system112 in response to detecting movement of the contacts described in FIGS.17A-17O on the touch-sensitive display system 112 while displaying theuser interfaces shown in FIGS. 17A-17O on the touch-sensitive displaysystem 112; in such embodiments, the focus selector is, optionally: arespective contact, a representative point corresponding to a contact(e.g., a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112, in place of cursor 11404.

FIGS. 17A-17G illustrate various embodiments where user interface 11408displays representations 11406 of a piece of music on display 450. Userinterface 11408 also displays cursor 11404, controlled by contact 11410on touch-sensitive surface 451 and movement 11412 thereof. In someembodiments, cursor 11404 moves over representation 11406 of a piece ofmusic, and in response, tactile generators 167 provide tactile feedback(e.g., tactile outputs 11414) that corresponds to at least a subset ofbeats in the piece of music (e.g., beats 11418). In some embodiments,after the tactile feedback has been provided, cursor 11404 moves awayfrom representation 11406 of a piece of music, and in response, tactilegenerators 167 cease to provide tactile feedback corresponding to beatsin the piece of music.

FIGS. 17A-17G illustrate that contact 11410, corresponding to cursor11404 displayed on display 450, and a gesture including movement 11412of contact 11410 (e.g., movement 11412-a of contact 11410 from location11410-a in FIG. 17A to location 11410-b in FIGS. 17B-17F and/or movement11412-b of contact 11410 from location 11410-b in FIGS. 17B-17F tolocation 11410-c in FIG. 17G) are detected on touch-sensitive surface451. Contact 11410 is detected at a position on touch-sensitive surface451 corresponding to an area on display 450 occupied by focus selector11404 (e.g., contact 11410 corresponds to a focus selector on thedisplay, such as cursor 11404 which is at or near a location of userinterface object 11402). In some embodiments, movement of contact 11410on touch-sensitive surface 451 corresponds to movement of focus selector(e.g., a cursor 11404) on display 450 (e.g., as illustrated in FIGS.17A-17G).

FIGS. 17A-17F illustrate various examples of a beginning of a gesturewhere cursor 11404 moves over representation 11406 of a piece of music,in accordance with movement 11412-a of contact 11410 on touch-sensitivesurface 451, corresponding to cursor 11404 on display 450. In FIGS.17B-17F, while focus selector 11404 remains over representation 11406 ofa piece of music, the device generates tactile outputs 11414 (e.g., viatactile output generators 167), corresponding to a subset of beats(e.g., beats 11418 shown in FIGS. 17C-17F) of the piece of music.

In some embodiments, as illustrated in FIGS. 17A-17G, the piece of musicis being played in a media player application (e.g., illustrated asmedia player application window 11402). In some embodiments, asillustrated in FIGS. 17A-17G, the representation of the piece of musicis a graphical representation of the piece of music (e.g., images 11406of cover art corresponding to an album of the piece of music).

FIGS. 17C-17F illustrate various embodiments where representation 11406of the piece of music is displayed in media player application window11402 and tactile feedback (e.g., tactile outputs 11414-1, 11414-3,11414-5, and 11414-7 in FIGS. 17C-17F) are generated when correspondingbeats (e.g., beats 11418-1, 11418-3, 11418-5, and 11418-7 in FIGS.17C-17F) in a subset of beats (e.g., a subset of all of the beats inpiece of music 11416) are played by the media player application. FIGS.17C-17F also illustrate various embodiments, where the subset of beats(e.g., those beats that correspond to tactile feedback provided to theuser) include stressed beats on every other beat, including the first(e.g., beat 11418-1), third (e.g., beat 11418-3), fifth (e.g., beat11418-5) and seventh (e.g., beat 11418-7) beats of the piece of music.

FIG. 17C illustrates an embodiment where the subset of beats excludesunstressed beats, including the second (e.g., beat 11418-2), fourth(e.g., beat 11418-4), sixth (e.g., beat 11418-6) and eighth (e.g., beat11418-8) beats of the piece of music (e.g., piece of music 11416). Incontrast, FIGS. 17D-17F, described below, illustrate variousembodiments, where the subset of beats includes both stressed beats(e.g., every odd beat 11418) and unstressed beats (e.g., every even beat11418) of the piece of music. FIG. 17D illustrates an embodiment, wherethe tactile outputs 11414 are substantially the same, regardless ofwhether they correspond to a stressed beat or an unstressed beat. Incontrast, FIGS. 17E-17F, described below, illustrate variousembodiments, where first tactile outputs 11414 corresponding to stressedbeats (e.g., odd number beats 11418) are substantially different fromsecond tactile outputs 11414 corresponding to unstressed beats (e.g.,even number beats 11418).

For example, FIG. 17E illustrates an embodiment where first tactileoutputs 11414 corresponding to stressed beats 11418 have a same orsubstantially same amplitude (e.g., high amplitude “A_(H)”) but asubstantially different movement profile (e.g., square waveform shape11436 as compared to sawtooth waveform shape 11434) as second tactileoutputs 11414 corresponding to unstressed beats 11418 of the piece ofmusic. In contrast, FIG. 17F illustrates an embodiment, where firsttactile outputs 11414 corresponding to stressed beats 11418 have asubstantially different amplitude (e.g., high amplitude “A_(H)” ascompared to low amplitude “A_(L)”) but substantially a same movementprofile (e.g., square waveform shape 11434) as second tactile outputs11414 corresponding to unstressed beats 11418 of the piece of music.

FIGS. 17B-17G illustrate various embodiments where after providingtactile feedback, as illustrated in FIGS. 17B-17F, the device detectsmovement 11412-b of contact 11410 on touch sensitive surface 451 thatcorresponds to movement of cursor 11404 away from representation 11406of the piece of music. As illustrated in FIG. 17G, in response to themovement of cursor 11404 away from representation 11406 of the piece ofmusic, tactile generators 167 cease providing tactile outputs 11414,corresponding to beats of the piece of music.

FIGS. 17H-17L illustrate various embodiments where user interface 11408displays music composition application window 11422, which includesrepresentation 11424 of a musical score, on display 450. User interface11408 also displays cursor 11404, controlled by contact 11426 ontouch-sensitive surface 451 and movement 11428 thereof. In someembodiments, cursor 11404 moves over representation 11424 of a musicalscore, and in response, tactile generators 167 provide tactile feedback(e.g., tactile outputs 11414) that corresponds to at least a subset ofbeats in the piece of music (e.g., beats 11418). In some embodiments,after the tactile feedback has been provided, cursor 11404 moves awayfrom representation 11424 of the musical score, and in response, tactilegenerators 167 cease to provide tactile feedback corresponding to beatsin the piece of music.

In some embodiments, as illustrated in FIGS. 17I-17J, movement of cursor11404 over representation 11424 of a musical score, in accordance withmovement 11428-a of contact 11426 on touch-sensitive surface 451, whilethe piece of music (e.g., a composition) is being played back by thecomposition application, results in the generation of tactile outputs11414 corresponding to beats 11418 of the piece of music when the beatsare played by the media application. Although FIG. 17J illustrates thatfirst tactile outputs 11414 corresponding to stressed beats 11418 (e.g.,the odd numbered tactile outputs 11414 and beats 11418, respectively)and second tactile outputs 11414 corresponding to unstressed beats 11418(e.g., the even numbered tactile outputs 11414 and beats 11418,respectively) feel substantially different (e.g., have a substantiallydifferent amplitude, but a same or substantially same square waveformmovement profile), in some embodiments, the first tactile outputs andsecond tactile outputs are generated with substantially the sameamplitude and movement profile and thus will feel substantially the sameto a user. In some embodiments, the second tactile outputs correspondingto unstressed beats are excluded (e.g., are not generated by thedevice).

In some embodiments, as illustrated in FIGS. 17K-17L, movement of cursor11404 over a representation 11430-3 of corresponding beat 11418-3 inmusical score 11424, in accordance with movement 11428-c of contact11426 on touch-sensitive surface 451, results in the generation oftactile feedback (e.g., tactile output 11414-3). For example, FIG. 17Killustrates an embodiment where cursor 11404, in accordance withmovement 11428-b of contact 11426 from position 11426-1 to position11426-3 on touch-sensitive surface 451, moves over representation 11424of a musical score on display 450 and tactile feedback is not generatedbecause the cursor is displayed at a position not corresponding to arepresentation of a beat in the piece of music. In contrast, asillustrated in FIG. 17L, in accordance with movement 11428-c of contact11426 from position 11426-3 to position 11426-d on touch-sensitivesurface 451, cursor 11404 moves over representation 11430-3 of beat11418-3 in representation 11424 of the musical score and tactilefeedback (e.g., tactile output 11414-3) is generated on touch-sensitivesurface 451, regardless of whether or not music corresponding torepresentation 11424 of the score of the piece of music is concurrentlybeing played by the music composition application.

FIGS. 17M-17O illustrate example waveforms of movement profiles forgenerating these tactile outputs. FIG. 17M illustrates a sawtoothwaveform 11434. FIG. 17N illustrates a square waveform 11436 and FIG.17O illustrates a square waveform 11438 that has a lower amplitude thanthe square waveform of FIG. 17F. Sawtooth waveform 11434 has a differentmovement profile from square waveforms 11436 and 11438 and substantiallythe same amplitude as square waveform 11436. Square waveform 11436 has asubstantially same movement profile and substantially differentamplitude than square waveform 11438.

FIGS. 18A-18B are flow diagrams illustrating a method 11500 of providingfeedback that corresponds to beats of a piece of music in accordancewith some embodiments. The method 11500 is performed at an electronicdevice (e.g., device 300, FIG. 3, or portable multifunction device 100,FIG. 1A) with a display and a touch-sensitive surface. In someembodiments, the display is a touch screen display and thetouch-sensitive surface is on the display. In some embodiments, thedisplay is separate from the touch-sensitive surface. Some operations inmethod 11500 are, optionally, combined and/or the order of someoperations is, optionally, changed.

As described below, the method 11500 provides an intuitive way toprovide feedback that corresponds to beats of a piece of music. Themethod reduces the cognitive burden on a user when detecting feedbackthat corresponds to beats of a piece of music, thereby creating a moreefficient human-machine interface. For battery-operated electronicdevices, enabling a user to detect feedback that corresponds to beats ofa piece of music faster and more efficiently conserves power andincreases the time between battery charges.

In some embodiments, the device displays (11502) a representation of apiece of music (e.g., representations 11406 of a piece of cover artcorresponding to a piece of music in FIGS. 17A-17G or representation11424 of a musical score corresponding to a piece of music in FIGS.17H-17L) on a display (e.g., display 450 in FIGS. 17A-17L). In someembodiments, the piece of music (e.g., music 11416 in FIGS. 17C-17F and17J) is played (11504) in a media player application (e.g., media playerapplication window 11402 in FIGS. 17A-17G), and the representation ofthe piece of music is a graphical representation of the piece of music(e.g., representations 11406 of a piece of cover art for an album of thepiece of music in FIGS. 17A-17G). In some embodiments, therepresentation of the piece of music is a piece of cover art for analbum of the piece of music (e.g., representations 11406). In someembodiments, the representation of the piece of music is a “now playing”region of the media player application (e.g., region 11420 of mediaplayer application window 11402) including a name and play time of thepiece of music.

In some embodiments, while the device displays the representation of apiece of music, the device detects (11506) movement of a focus selector(e.g., cursor 11404 in FIGS. 17A-17L) over the representation of thepiece of music.

In some embodiments, while detecting the focus selector over therepresentation of the piece of music, the device provides (11508)tactile feedback (e.g., tactile outputs 11414 in FIGS. 17B-17F, 17I-17Jand 17L) that corresponds to at least a subset of beats (e.g., beats11418 in FIGS. 17C-17F and 17J) of the piece of music (e.g., piece ofmusic 11416 in FIGS. 17C-17F and 17J). In some embodiments, therepresentation of the piece of music is a representation of the musicalnotes (e.g., a musical score). In some embodiments, the representationof the piece of music includes visual media corresponding to the pieceof music (e.g., an image, a video, a text description of the piece ofmusic or an album/video including the piece of music, or an audiovisualizer associated with the piece of music). For example, the tactilefeedback corresponding to a piece of music is generated while a focusselector (e.g., a displayed cursor or a contact) is over an album coverfor the piece of music, over a composer or artist image for the piece ofmusic, or over a currently playing video that includes the piece ofmusic as currently playing audio content (e.g., the background music ina movie or the music in a music video).

In some embodiments, after providing the tactile feedback, the devicedetects (11528) movement of the focus selector away from therepresentation of the piece of music (e.g., movement of cursor 11404,corresponding to movement 11412-b of contact 11410-c on touch-sensitivesurface 451, in FIG. 17G).

In some embodiments, in response to detecting movement of the focusselector away from the representation of the piece of music, the deviceceases (11530) to provide the tactile feedback (e.g., tactile outputs11414) that corresponds to the beats of the piece of music.

In some embodiments, the focus selector (e.g., cursor 11404 in FIGS.17A-17L) is moved (11510) in accordance with movement (e.g., movements11412 in FIGS. 17A-17G or movements 11428 in FIGS. 17H-17L) of a contact(e.g., contact 11410 in FIGS. 17A-17G or contact 11426 in FIGS. 17H-17L)on a touch-sensitive surface (e.g., touch-sensitive surface 451), andthe tactile feedback is provided by generating tactile outputs (e.g.,tactile outputs 11414 in FIGS. 17B-17F, 17I-17J and 17L) on thetouch-sensitive surface. In some embodiments, the contact is the focusselector (e.g., when the device has a touch screen, the focus selectoris, optionally, contact 11410). In some embodiments, the contactcorresponds to a cursor or selection box that is displayed on thedisplay.

In some embodiments, the representation of the piece of music (e.g.,representations 11406 of a piece of cover art for an album of the pieceof music in FIGS. 17A-17G) is displayed (11512) in a media playerapplication (e.g., media player application window 11402 in FIGS.17A-17G), and the tactile feedback includes a plurality of tactileoutputs (e.g., tactile outputs 11414) generated when corresponding beats(e.g., beats 11418) in the subset of beats are played by the mediaplayer application (e.g., the touch-sensitive surface generates tactileoutputs in time with music being played in the media playerapplication).

In some embodiments, the representation of the piece of music isdisplayed (11514) as a musical score (e.g., representation 11424 of ascore of the piece of music in FIGS. 17H-17L) in a music composingapplication (e.g., music composition application window 11422 in FIGS.17H-17L). For example, by displaying a representation of notes (e.g.,black bar 11430-9, corresponding to the note played at beat 11418-9 ofpiece of music 11416, in FIGS. 17H-17L) of the piece of music (e.g.,music 11416) in a representation of a musical score (e.g.,representation 11424 of a musical score) corresponding to the piece ofmusic.

In some embodiments, while the representation of the piece of music isdisplayed as a musical score in a music composing application, thetactile feedback includes (11516) a plurality of tactile outputs (e.g.,tactile outputs 11414) generated when the focus selector moves overrepresentations of corresponding beats (e.g., beat 11418-3 representedas vertical line 11432-3 or beat representation 11430-3 in FIG. 17L) inthe subset of beats in the musical score. In some embodiments, the focusselector moves in accordance with movement of a contact on thetouch-sensitive surface (e.g., movement 11428-c of contact 11426 ontouch-sensitive surface 451 in FIGS. 17K-17L), and thus the tactileoutputs (e.g., tactile outputs 11414) are generated in accordance withmovement of the contact on the touch-sensitive surface.

In some embodiments, the subset of beats (e.g., beats 11418) includes(11518) stressed beats in the piece of music (e.g., even numbered beats11418 in piece of music 11416 in FIGS. 17C-17F and 17J). In someembodiments, a beat is the basic unit of time in music (e.g., a quarternote in a piece of music having a 4/4 time signature or an eighth notein a piece of music having a 6/8 time signature), where a stressed beatis a stronger, louder or otherwise more emphatic beat of a plurality ofbeats. Some typical beat patterns include stressing every fourth beat(e.g., as commonly done in music having a 4/8 time signature), stressingevery other beat (e.g., as commonly done in music having a 4/4 timesignature) or stressing every third beat (e.g., as commonly done inmusic having a 3/4 or 6/8 time signature, such as a waltz). A beat thatis not stressed is sometimes referred to as an unstressed beat. In someembodiments, a beat is a subunit of the basic unit of time in music(e.g., an eighth note in music having a 4/4 or 3/4 time signature).

In some embodiments, the subset of beats excludes (11520) unstressedbeats of the piece of music. For example, as illustrated in FIG. 17C,the subset of beats 11418 includes stressed beats (e.g., odd numberedbeats 11418) but excludes unstressed beats (e.g., even numbered beats11418), and tactile outputs 11414 are only generated corresponding tothe stressed beats.

In some embodiments, where tactile feedback corresponding to at least asubset of beats of the piece of music is provided (11508) whiledetecting the focus selector over the representation of the piece ofmusic, the subset of beats include (11522) one or more stressed beats(e.g., odd numbered beats 11418) and one or more unstressed beats (e.g.,even numbered beats 11418), the tactile feedback includes a plurality oftactile outputs (e.g., tactile outputs 11414) for corresponding beats inthe subset of beats, a first tactile output is generated for stressedbeats, and a second tactile output, different from the first tactileoutput, is generated for non-stressed beats (e.g., even numbered tactileoutputs 11414 and odd numbered tactile outputs 11414 feel substantiallydifferent to the user, as represented in FIGS. 17E-17F and 17J). Incontrast, in some embodiments, a first tactile output corresponding to astressed beat and a second tactile output corresponding to an unstressedbeat are substantially the same (e.g., odd numbered tactile outputs11414 corresponding to odd numbered stressed beats 11418 and evennumbered tactile outputs 11414 corresponding to even numbered unstressedbeats 11418 feel substantially the same to the user, as represented inFIG. 17D). In some embodiments, the first tactile output is moreprominent (e.g., has a larger amplitude) than the second tactile output.In some embodiments, the second tactile output is more prominent (e.g.,has a larger amplitude) than the first tactile output.

In some embodiments, the first tactile output is generated (11524) bymovement of the touch-sensitive surface that includes a first dominantmovement component, the second tactile output is generated by movementof the touch-sensitive surface that includes a second dominant movementcomponent, and the first dominant movement component and the seconddominant movement component have a same or substantially same amplitude(e.g., high amplitude “A_(H)” of all tactile outputs 11414 in FIG. 17E)and substantially different movement profiles (e.g., square waveformshape 11436 of odd numbered tactile outputs 11414 and sawtooth waveformshape 11434 of even numbered tactile outputs 11414 in FIG. 17E). In someembodiments, movement of the touch-sensitive surface corresponds to aninitial impulse, ignoring any unintended resonance. In some embodiments,the movement profiles differ in their waveform shape (e.g., square,sine, squine, triangle or sawtooth waveform shape), waveform pulse widthand/or waveform pulse period (e.g., frequency). For example, asillustrated in FIG. 17E, a “detent” that is generated on thetouch-sensitive surface corresponding to a stressed beat of the musichas a square waveform movement profile (e.g., square waveform 11436 ofodd numbered tactile outputs 11414 in FIG. 17E), whereas a “click” thatis generated on the touch-sensitive surface corresponding to anunstressed beat of the music has a sawtooth waveform movement profile(e.g., sawtooth waveform 11434 of even numbered tactile outputs 11414 inFIG. 17E), or vice versa.

In some embodiments, the first tactile output is generated (11526) bymovement of the touch-sensitive surface that includes a first dominantmovement component, the second tactile output is generated by movementof the touch-sensitive surface that includes a second dominant movementcomponent, and the first dominant movement component and the seconddominant movement component have a same or substantially same movementprofile (e.g., square waveforms 11434 of odd numbered tactile outputs11414 and square waveform 11436 of even numbered tactile outputs 11414in FIGS. 17F and 17J) and substantially different amplitudes (e.g., highamplitude “A_(H)” of odd numbered tactile outputs 11414 is greater thanlow amplitude “A_(L)” of even numbered tactile outputs 11414 in FIGS.17F and 17J). In some embodiments, movement of the touch-sensitivesurface corresponds to an initial impulse, ignoring any unintendedresonance. In some embodiments, the movement profiles differ in theirwaveform shape (e.g., square, sine, squine, triangle or sawtoothwaveform shape), waveform pulse width and/or waveform pulse period(e.g., frequency). For example, as illustrated in FIGS. 17F and 17J, a“detent” that is generated on the touch-sensitive surface correspondingto a stressed beat of the music has a greater amplitude than a “detent”that is generated on the touch-sensitive surface corresponding to anunstressed beat of the music (e.g., high amplitude “A_(H)” of oddnumbered tactile outputs 11414 in FIGS. 17F and 17J is greater than lowamplitude “A_(L)” of even numbered tactile output 11414 in FIGS. 17F and17J), or vice versa.

In some embodiments, after providing tactile feedback, the devicedetects (11528) movement of the focus selector away from therepresentation of the piece of music. For example, as illustrated inFIG. 17G, in accordance with detection of movement 11412-b of contact11410 from position 11410-b to position 11410-c on touch-sensitivesurface 451, corresponding to movement of cursor 11404 away fromrepresentation 11406 of a piece of music. In some embodiments, inresponse to detecting movement of the focus selector away from therepresentation of the piece of music, the device ceases (11530) toprovide tactile feedback that corresponds to the beats of the piece ofmusic. For example, as illustrated in FIG. 17G, when cursor 11404 movesaway from representation 11406 of a piece of music, tactile outputgenerators 167 stop generating tactile outputs 11414 on touch-sensitivesurface 451 because the cursor is no longer positioned over therepresentation of the piece of music.

It should be understood that the particular order in which theoperations in FIGS. 18A-18B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein are alsoapplicable in an analogous manner to method 11500 described above withrespect to FIGS. 18A-18B. For example, the contacts, gestures, userinterface objects, tactile sensations and focus selectors describedabove with reference to method 11500 optionally have one or more of thecharacteristics of the contacts, gestures, user interface objects,tactile sensations and focus selectors described herein with referenceto other methods described herein. For brevity, these details are notrepeated here.

In accordance with some embodiments, FIG. 19 shows a functional blockdiagram of an electronic device 11600 configured in accordance with theprinciples of the various described embodiments. The functional blocksof the device are, optionally, implemented by hardware, software, or acombination of hardware and software to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 19 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As shown in FIG. 19, an electronic device 11600 includes a display unit11602 configured to display one or more user interface objects, atouch-sensitive surface unit 11604 configured to receive user contacts,optionally one or more sensor units 11606 configured to detect intensityof contacts with the touch-sensitive surface unit 11604; and aprocessing unit 11608 coupled to the display unit 11602, thetouch-sensitive surface unit 11604 and optionally the one or more sensorunits 11606. In some embodiments, the processing unit 11608 includes adisplay enabling unit 11610, a detecting unit 11612, a providing unit11614, and a ceasing unit 11616.

In some embodiments, the processing unit 11608 is configured to enabledisplay (e.g., with the display enabling unit 11610) of a representationof a piece of music on display unit 11602. In some embodiments, theprocessing unit 11608 is configured to detect movement of a focusselector over the representation of the piece of music (e.g., withdetecting unit 11612); and while detecting the focus selector over therepresentation of the piece of music, the processing unit 11608 isconfigured to provide tactile feedback that corresponds to at least asubset of beats of the piece of music (e.g., with providing unit 11614).In some embodiments, after providing the tactile feedback, theprocessing unit 11608 is configured to detect movement of the focusselector away from the representation of the piece of music (e.g., withthe detecting unit 11612); and in response to detecting movement of thefocus selector away from the representation of the piece of music, theprocessing unit 11608 is configured to cease to provide the tactilefeedback that corresponds to the beats of the piece of music (e.g., withthe ceasing unit 11616).

In some embodiments, the processing unit 11608 is configured to enabledisplay of movement of the focus selector (e.g., with the displayenabling unit 11610) in accordance with movement of a contact ontouch-sensitive surface unit 11604, and the tactile feedback is providedby generating tactile outputs on the touch-sensitive surface unit 11604(e.g., with the providing unit 11614).

In some embodiments, the piece of music is currently being played in amedia player application; and the representation of the piece of musicis a graphical representation of the piece of music.

In some embodiments, the processing unit 11608 is configured to displaythe representation of the piece of music in a media player application(e.g., with the display enabling unit 11610), and the tactile feedbackincludes a plurality of tactile outputs generated when correspondingbeats in the subset of beats are played by the media player application.

In some embodiments, the processing unit 11608 is configured to displaythe representation of the piece of music as a musical score in a musiccomposing application (e.g., with the display enabling unit 11610).

In some embodiments, the tactile feedback includes a plurality oftactile outputs generated when the focus selector moves overrepresentations of corresponding beats in the subset of beats in themusical score.

In some embodiments, the subset of the beats includes stressed beats ofthe piece of music.

In some embodiments, the subset of the beats excludes unstressed beatsof the piece of music.

In some embodiments, the subset of beats include one or more stressedbeats and one or more unstressed beats, the tactile feedback includes aplurality of tactile outputs for corresponding beats in the subset ofbeats, the processing unit 11608 is configured to generate a firsttactile output for stressed beats (e.g., with the providing unit 11614),and the processing unit 11608 is configured to generate a second tactileoutput, different from the first tactile output, for non-stressed beats(e.g., with the providing unit 11614).

In some embodiments, the first tactile output is generated by movementof the touch-sensitive surface unit 11604 that includes a first dominantmovement component, the second tactile output is generated by movementof the touch-sensitive surface unit 11604 that includes a seconddominant movement component, and the first dominant movement componentand the second dominant movement component have a same amplitude anddifferent movement profiles.

In some embodiments, the first tactile output is generated by movementof the touch-sensitive surface unit 11604 that includes a first dominantmovement component, the second tactile output is generated by movementof the touch-sensitive surface unit 11604 that includes a seconddominant movement component, and the first dominant movement componentand the second dominant movement component have a same amplitude anddifferent movement profiles.

The operations in the information processing methods described aboveare, optionally implemented by running one or more functional modules ininformation processing apparatus such as general purpose processors(e.g., as described above with respect to FIGS. 1A and 3) or applicationspecific chips.

The operations described above with reference to FIGS. 18A-18B are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.19. For example, detection operations 11506 and 11528 and providingoperation 11508 are, optionally, implemented by event sorter 170, eventrecognizer 180, and event handler 190. Event monitor 171 in event sorter170 detects a contact on touch-sensitive display 112, and eventdispatcher module 174 delivers the event information to application136-1. A respective event recognizer 180 of application 136-1 comparesthe event information to respective event definitions 186, anddetermines whether a first contact at a first location on thetouch-sensitive surface corresponds to a predefined event or sub-event,such as selection of an object on a user interface or display of a focusselector over a representation of a piece of music on a user interface.When a respective predefined event or sub-event is detected, eventrecognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionallyutilizes or calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. In some embodiments, event handler 190 accesses arespective tactile output generator 167 to generate a tactile output.Similarly, it would be clear to a person having ordinary skill in theart how other processes can be implemented based on the componentsdepicted in FIGS. 1A-1B.

It should be understood that the particular order in which theoperations have been described above is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that the various processes separatelydescribed herein can be combined with each other in differentarrangements. For example, the contacts, user interface objects, tactilesensations, intensity thresholds, and/or focus selectors described abovewith reference to any one of the various processes separately describedherein optionally have one or more of the characteristics of thecontacts, gestures, user interface objects, tactile sensations,intensity thresholds, and focus selectors described herein withreference to one or more of the other methods described herein. Forbrevity, all of the various possible combinations are not specificallyenumerated here, but it should be understood that the claims describedabove may be combined in any way that is not precluded by mutuallyexclusive claim features.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the various described embodiments to the precise forms disclosed.Many modifications and variations are possible in view of the aboveteachings. The embodiments were chosen and described in order to bestexplain the principles of the various described embodiments and theirpractical applications, to thereby enable others skilled in the art tobest utilize the various described embodiments with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A non-transitory computer readable storage mediumstoring one or more programs, the one or more programs comprisinginstructions, which when executed by an electronic device with adisplay, a touch-sensitive surface, cause the device to: display, on thedisplay, respective content that includes an image; detect a gesture onthe touch-sensitive surface for adjusting an image size of the image;and in response to detecting the gesture: determine an adjustment to theimage size that corresponds to an extent of the gesture; in accordancewith a determination that the adjustment to the image size would causethe image size to exceed a predefined image sizing limit: generate arespective tactile output on the touch-sensitive surface; perform theadjustment to the image size and display the image at a sizecorresponding to the adjustment to the image size; and in response todetecting liftoff of a contact used to perform the gesture, at leastpartially reverse the adjustment to the image size, such that the imagesize is reduced to a size corresponding to the predefined image sizinglimit, and display the image at a size corresponding to the predefinedimage sizing limit; and in accordance with a determination that theadjustment to the image size would not cause the image size to exceedthe predefined image sizing limit, perform the adjustment to the imagesize without generating the respective tactile output on thetouch-sensitive surface, and display the image at a size correspondingto the adjustment to the image size.
 2. The non-transitory computerreadable storage medium of claim 1, wherein: adjusting the image sizeincludes adjusting a cropping mask; the predefined image sizing limitcomprises a content boundary; and the content boundary is exceeded whenthe cropping mask extends beyond the content boundary.
 3. Thenon-transitory computer readable storage medium of claim 1, wherein: theimage has a shape; and the predefined image sizing limit comprises amaximum allowable adjustment to a size of the shape.
 4. Thenon-transitory computer readable storage medium of claim 1, wherein thedisplayed image, prior to the electronic device detecting the gesture onthe touch-sensitive surface for adjusting an image size of the image,has an image size that is an initial image size smaller than thepredefined image sizing limit.
 5. An electronic device, comprising: adisplay; a touch-sensitive surface; one or more processors; memory; andone or more programs, wherein the one or more programs are stored in thememory and configured to be executed by the one or more processors, theone or more programs including instructions for: displaying, on thedisplay, respective content that includes an image; detecting a gestureon the touch-sensitive surface for adjusting an image size of the image;and in response to detecting the gesture: determining an adjustment tothe image size that corresponds to an extent of the gesture; inaccordance with a determination that the adjustment to the image sizewould cause the image size to exceed a predefined image sizing limit:generating a respective tactile output on the touch-sensitive surface;performing the adjustment to the image size and displaying the image ata size corresponding to the adjustment to the image size; and inresponse to detecting liftoff of a contact used to perform the gesture,at least partially reversing the adjustment to the image size, such thatthe image size is reduced to a size corresponding to the predefinedimage sizing limit, and displaying the image at a size corresponding tothe predefined image sizing limit; and in accordance with adetermination that the adjustment to the image size would not cause theimage size to exceed the predefined image sizing limit, performing theadjustment to the image size without generating the respective tactileoutput on the touch-sensitive surface, and displaying the image a a sizecorresponding to the adjustment to the image size.
 6. The electronicdevice of claim 5, wherein: adjusting the image size includes adjustinga cropping mask; the predefined image sizing limit comprises a contentboundary; and the content boundary is exceeded when the cropping maskextends beyond the content boundary.
 7. The electronic device of claim5, wherein: the image has a shape; and the predefined image sizing limitcomprises a maximum allowable adjustment to a size of the shape.
 8. Theelectronic device of claim 5, wherein the displayed image, prior to theelectronic device detecting the gesture on the touch-sensitive surfacefor adjusting an image size of the image, has an image size that is aninitial image size smaller than the predefined image sizing limit.
 9. Amethod, comprising: at an electronic device with a touch-sensitivesurface and a display: displaying, on the display, respective contentthat includes an image; detecting a gesture on the touch-sensitivesurface for adjusting an image size of the image; and in response todetecting the gesture: determining an adjustment to the image size thatcorresponds to an extent of the gesture; in accordance with adetermination that the adjustment to the image size would cause theimage size to exceed a predefined image sizing limit: generating arespective tactile output on the touch-sensitive surface; performing theadjustment to the image size and displaying the image at a sizecorresponding to the adjustment to the image size; and in response todetecting liftoff of a contact used to perform the gesture, at leastpartially reversing the adjustment to the image size, such that theimage size is reduced to a size corresponding to the predefined imagesizing limit, and displaying the image at a size corresponding to thepredefinded image sizing limit; and in accordance with a determinationthat the adjustment to the image size would not cause the image size toexceed the predefined image sizing limit, performing the adjustment tothe image size without generating the respective tactile output on thetouch-sensitive surface, and displaying the image at a sizecorresponding to the adjustment to the image size.
 10. The method ofclaim 9, wherein: adjusting the image size includes adjusting a croppingmask; the predefined image sizing limit comprises a content boundary;and the content boundary is exceeded when the cropping mask extendsbeyond the content boundary.
 11. The method of claim 9, wherein: theimage has a shape; and the predefined image sizing limit comprises amaximum allowable adjustment to a size of the shape.
 12. The method ofclaim 9, wherein the displayed image, prior to the device detecting thegesture on the touch-sensitive surface for adjusting an image size ofthe image, has an image size that is an initial image size smaller thanthe predefined image sizing limit.